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Preface 


We are pleased you have chosen to enjoy the 1978 Na¬ 
tional Computer Conference (NCC). Frequently the NCC is 
judged to be “too large” to have a theme. This year we 
bring you two themes: the primary theme is the national 
energy problem and how computers can contribute to its 
alleviation; the secondary theme is the growth of the com¬ 
puter industry, especially its fastest growing segment, per¬ 
sonal computing. 

The energy and computation theme is highlighted in the 
keynote address and in a powerful panel for the Tuesday 
evening’s Symposium on Developing Energy and Computing 
Technology. It is continued by the luncheon speakers and 
by more than 10 percent of the Technical and Professional 
Program. 

On coping with growth in the computer industry, a portion 
of the Technical and Professional Program deals with per¬ 
sonal computing and with developing education curricula. 
The principal highlight is the Personal Computing Festival, 
with exhibits, sessions, and contests all its own. This is the 
first time the National Computer Conference has had sepa¬ 
rate registration with no age limit on attendance for a portion 
of its program. 


This document forms the principal permanent record of 
the Technical and Professional Program of this conference. 
As such, it is a vital part of the documentation and infor¬ 
mation-dissemination system in our community. The explo¬ 
sive growth of literature in the last decade makes it imper¬ 
ative that this dissemination system be examined carefully. 
One such examination has been conducted by Thomas J. 
Allan, in his interesting NSF-funded, ten-year study of this 
information-dissemination system and reported in The Man¬ 
agement of Technology Flow (MIT 1977). Most such studies 
lump scientific and engineering literature; however, he 
draws a careful distinction between them. One part of that 
distinction is that a given topic generally appears earlier in 
time in scientific literature than in engineering literature. 
During this time delay three very important things occur: 
diffusion of the information through the community; trans¬ 
lation into the language of current technical practice; and 
evaluation of the concepts. This evaluation tends to be re¬ 
lated to enterprise objectives and, hence, of strong interest 
to technical management. 

Despite the explosive growth in the number of confer¬ 
ences, symposia, seminars, and workshops dealing with the 
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computer industry, the National Computer Conference has 
remained the premier conference to enhance this diffusion, 
speed this translation, and sharpen this evaluation. NCC is 
supported by 15 individual technical societies which provide 
it with the multidisciplinary support that makes it the focal 
point of excellence in the literature for technology and man¬ 
agement. 

To this end, in 1978 we have not equated originality with 
quality. Rather, we have brought the best judgment of a 
very strong technical and professional program committee 
together to ensure that the topics of importance to the com¬ 
munity at large are represented. The best of these topics are 
presented in either paper or panel form. You will note there 
are two papers in this proceedings that have been presented 
earlier for a smaller audience. “The art of artificial intelli¬ 
gence—Themes and case studies of knowledge engineer¬ 
ing,” by Dr. Edward A. Feigenbaum and “The ubiquity of 
discovery,” by Dr. Douglas B. Lenat. Within the AI com¬ 


munity these have been judged “best papers” and hence are 
repeated here to speed the translation process and to en¬ 
hance the diffusion throughout the larger information proc¬ 
essing community. 

Likewise, we have attempted to present both sides of 
controversial issues so that society itself can participate in 
the evaluation. Only in a large multidisciplinary forum of 
this type can we begin to provide approaches to solutions 
of the very pervasive problems, such as the national energy 
problem, the facets of a public information policy, the prob¬ 
lems of federal regulation, etc., that face us today. 

My undying thanks go to the literally hundreds of people 
whose efforts have gone into bringing you this fine confer¬ 
ence, but especially to those who have dedicated their time, 
talent, and effort to serve on its many committees. The 
names of most of them are acknowledged in the back of this 
proceedings. 




LEONARD Y. LIU 
Program Co-Chairman 
IBM Corporation 
San Jose, California 


SAKTI P. GHOSH 
Program Co-Chairman 
IBM Corporation 
San Jose, California 


Introduction 


SHAPING THE PROGRAM 

Of the many criteria that were followed in creating the 
technical program committee of 1978, the one that played 
the dominant role was “excellence.” We believed that at¬ 
tracting top people to participate in the technical program 
committee would result in an excellent technical program. 
We were indeed fortunate in attracting top technical and 
professional people from industry, government, universities, 
and other parts of society which are associated with the 
computer industry. We also included in our committee top 
professionals from our sponsoring societies; IEEE Com¬ 
puter Society, ACM, SCS, and DPMA. Cooperation from 
each of the non-sponsoring constituent societies of AFIPS 
was requested through their Presidents by our conference 
vice-chairman. 

The program committee in its first meeting in December 
1976 established the following philosophy and aims: (i) The 
number and reactions of attendees are the major criteria for 
measuring the success of the conference; (ii) Seek broad 
base appeal—do not concentrate on applications for indus¬ 
trial usage and ignore the system questions. Do not concen¬ 
trate on the systems architecture problems and ignore the 


general public; (iii) Orient the program toward industrial and 
commercial audiences. With this charter in hand, the tech¬ 
nical program committee began identifying appropriate tech¬ 
nical areas for the program and created a broadly structured 
program with built-in adaptability to accomodate unsolicited 
high quality papers. 


PROGRAM STRUCTURE 

The program is structured into four major areas: Appli¬ 
cations, Methodology, Systems and People and Society. 
Each major area is subdivided into multiple basic areas. 
Each member of the program committee was responsible for 
at least one basic area. Each basic area was headed by an 
area director; some area directors were also a member of 
the program committee. The area directors were selected 
for both their professional reputation and motivation. The 
basic areas cover most of the traditional important aspects 
of computer knowledge. We have tried to capture some 
emerging important areas such as Computer Models in Solv¬ 
ing World’s Energy Problem, Electronic Fund Transfers, 
Special Purpose Terminals, Office Automation, Home and 
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Hobbying Computing, Evolution of New Hardware Tech¬ 
nology, and Legislation and Its Impact. For the first time in 
NCC’s history, we have asked a sister nation, namely Japan, 
to organize an area in our conference. 

One of the primary responsibilities of the area directors 
was to solicit high quality papers. The rule “every paper 
will be refereed” was applied universally to all solicited and 
unsolicited papers with the exception of two invited papers. 
The program contains many panel sessions. Each panelist 


was asked to prepare a position paper and have it approved 
by the session chairperson. In some cases the position pa¬ 
pers of panelists participating in a session were combined 
into a panel paper and are included in these proceedings. 
Most of the area directors have included an overview of 
their area. 

We hope you will find these proceedings of the program 
valuable and rewarding for many years to come. 



PART I—APPLICATIONS 




Area Director: 

William F. Rousseau 
Lawrence Livermore Laboratory 
Livermore, California 


Computer models in solving the world’s energy problems 


The '73-'74 Oil Embargo and the more recent shortages of natural gas have 
made the general public energy conscious. Moreover, depleted oil and gas re¬ 
sources, energy conservation legislation, and higher energy prices are creating 
problems in industry. Significant government programs have been initiated; more, 
possibly much more, is yet to come. However, causes and cures of our energy 
problems are controversial. Complexity of energy supply and use play as big a 
role in this confusion as differences in economic and political philosophy. The 
complexity makes it difficult to accurately assess the impact of corrective action 
or even if any is needed and is forcing many policy analysts to turn to computer 
models for help in making major public and private energy policy decisions. More 
conventional applications of computers to energy are also expanding with the 
objectives of finding or recovering more oil and gas or making energy systems 
more efficient. 

These sessions on the applications of computers in solving the worlds energy 
problems can only sample the many possibilities. Two sessions, “Energy Deci¬ 
sion Analysis” and “Energy Modeling Panel,” are devoted to the very significant 
application of computer modeling techniques to energy policy analysis. Two 
other sessions, “Computers in Petroleum Exploration” and “Computers in En¬ 
ergy Technology,” provide examples from the technological and engineering 
side. 

“Energy Decision Analysis,” chaired by Robert Karsan, has papers by working 
policy analysts. Tani’s paper reviews an analysis of President Ford’s proposed 
synthetic fuels commercialization program that seemed to have had a major 
impact—to kill the program. Brock’s paper addresses the economic theory used 
to construct several of the large energy models. Schweizer describes a model 
developed to predict peak electric load demand—a very important quantity in 
planning electric power system expansion and energy use. Rice and Meeske 
describe a model for natural gas demand forecasting based on interfuel compe¬ 
tition. Although computers are not emphasized in these papers, the work de¬ 
scribed would not be practical without them. 

“Computers in energy technology” chaired by Julius Chang, provides exam¬ 
ples of computer use to improve existing technology. Kamel and Wolf show how 
computers are being applied to build safer and more energy efficient automobiles. 
Westbrock and Haselman describe work on modeling what goes on in the cylinder 


1 



2 


National Computer Conference, 1978 


on an internal combustion engine for computer simulation. Price presents a model 
which relates to improved oil recovery from existing wells. 

“Computers in Petroleum exploration,’’ chaired by Pierre L. Goupillaud, has 
papers on several aspects of oil and gas exploration. Hoffman and Ternus discuss 
application of computers to petroleum exploration in general. Savit predicts 
enormous data processing requirements for geophysical data. Selzler gets into 
seismic modeling. 

“Energy Modeling Panel,” chaired by William F. Rousseau, addresses several 
significant questions raised by the widespread use of energy models in policy 
analysis. Participants all have wide experience with energy modeling and policy 
analysis. Questions include why the models were constructed, what questions 
they answer, historically the significant insights they have provided, how the 
models are being judged for credibility, and the extent to which these models are 
meaningful representations of the real world. 



Forecasting national gas demand by 
modeling fuel purchasing decisions 
for end-use customer groups 

by THOMAS R. RICE 

Applied Decision Analysis 
Palo Alto, California 

and 

C. JOHN MEESKE 

Private Consultant 
Troy, Michigan 


INTRODUCTION 

Rapidly rising domestic natural gas prices and the develop¬ 
ment of higher-priced synthetic fuels have made marketa¬ 
bility a key issue for the natural gas industry, both for 
distribution companies and for producers and interstate 
pipelines in making reserve acquisition decisions. This paper 
describes the methodology underlying a detailed model that 
characterizes the interfuel competition between gas, oil, 
coal, purchased steam, and electricity. 

The model forecasts the quantity of various fuels de¬ 
manded in a utility company’s service territory by consid¬ 
ering the fuel-purchasing decisions of customer groups. 
Each market is segmented into customer groups according 
to equipment types and operating characteristics. For ex¬ 
ample, residential customers are divided according to their 
heating system (forced air systems, gravity systems, boilers, 
and heat pumps). Commercial customers are divided by 
Standard Industrials Classification (SIC) codes such as res¬ 
taurants, laundries, schools, and hospitals. The industrial 
customers are also divided by SIC codes, and further sub¬ 
divided by major installations. In some cases, both com¬ 
mercial and industrial customers are further broken down 
by actual device. For example, large boilers at a major 
industrial power plant form a separate market segment. 

Consumers switch from one fuel type to another when it 
becomes economically desirable to do so, based on a net 
present value calculation that takes into account the follow¬ 
ing factors: 

• “hard” economic costs associated with each fuel type 

• subjective preferences for various fuels 

• availability of alternate fuels 

• efficiency of each fuel in providing usable energy 

• relative importance of present and future costs to each 
consumer 


• effect of government conservation programs and regu¬ 
lations 

• risk of fuel shortages 

Alternate gas rate designs included in the model provide 
for measurement of the impact of gas rates on gas demand. 
The model allows customers to switch to alternate (or mul¬ 
tiple) fueled equipment when it is technically possible and 
economically desirable, taking into account the possibility 
that a consumer may be technically or contractually required 
to use a minimum amount of fuel with such equipment. The 
forecasted demand for each fuel type is produced by sum¬ 
ming the demand for that fuel type for each consumer sector. 


MODEL OVERVIEW 

The major elements of the Interfuel Competition Model 
are shown in Figure 1. The central logic of the model deals 
with the manner in which customers expand the capacity of 
their energy consuming equipment and with their choice of 
fuel for both new and existing equipment. This central logic 
is supplied with information from several submodels. The 
Gas Rate Model computes the retail price of gas by sector 
based on projected cost of gas, sales volume, and distribu¬ 
tion costs. Alternate fuel prices for coal, oil, and electricity 
depend on sulfur content, annual usage, and proximity to 
rail and water transportation. The Industry Growth Model 
considers economic growth by SIC code, the corresponding 
impact upon energy growth, and the relative growth of dif¬ 
ferent technologies within a sector. 

The Capacity Growth Model contains logic to deal with 
new energy consuming capacity added to keep pace with 
changes in economic activity level. It calculates the amount 
of energy consuming capacity existing, added, retired, and 
refurbished within each sector for each year under consid- 
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Figure 1—Overview of interfuel competition model 


eration. Refurbishment is important since it represents a 
point where the incremental cost of converting fuel types is 
low. 

The heart of the Interfuel Competition Model is the 
Switching Logic shown in Figure 1. This logic analyzes 
consumer decisions regarding possible fuels to meet energy 
needs during each of the 20 years considered in the study. 
The customer’s decision is based on minimizing the net 
present cost associated with each possible fuel, including 
the cost of converting from one fuel to another as well as 
future prices of various fuels. Net present cost (NPC) is one 
component of the more general quantity, net present value 
(NPV). The net present value of a stream of investments, 
returns, and variable costs at some time period t is given 
by: 

n r? .-vc 

NPV t =(l-INCOME TAX RATE)Y 

j=t (l+r 

N T 

-(1-INVEST. TAX CREDIT) £ (D 

where Ij is any investment made in time period j, R j is the 
gross return or revenue in period j, VC, is the variable cost 
in period j, r is the discount rate, and N is the decision 
maker’s planning horizon. 

This equation states that the net present value is equal to 
the net after-tax value in each period (revenue minus any 
investments, all weighted by the appropriate tax rates), dis¬ 
counted by the appropriate rate and summed over all time 
periods. The equation for the net present value can be writ¬ 
ten as the difference of two quantities, the net present return 
(NPR) and the net present cost (NPC): 

NPV^NPRt-NPQ (2) 

The net present return is equal to the sum of the dis¬ 


counted returns in each time period weighted by 1 minus the 
income tax rate: 

N n 

NPRe=(l -INCOME TAX RATE) 2 (3) 

The net present cost is equal to the discounted variable 
costs and investments in each time period, weighted by the 
appropriate tax rates: 

N yC 

NPQ=(1-INCOME TAX RATE) £ 7 - 

i=t (l+r) J * 

N 1 

+(1 -INVEST. TAX CREDIT) £ 7T T~ V 7 ( 4 ) 
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If a consumer switches from one fuel to another and 
continues to use the same amount of energy, the benefit or 
return that he derives from that energy in each period will 
be unchanged. Thus, the net present return does not depend 
on the fuel chosen.* 

However, switching from one fuel to another will probably 
require a capital investment determined by the type of fuel 
selected; the variable costs associated with each fuel will be 
different. 

The variable cost associated with using a particular fuel 
in time period j can be written as follows: 

VC j =[P j (\+CONVENIENCE PREMIUM) 

+(0&M) j ]/FUEL EFFICIENCY (5) 

where Pj is the price of the fuel in time period j, (0&M)j is 
the operations and maintenance cost associated with using 
the fuel in time period j. “Convenience Premium” is a sub¬ 
jective term used to scale the actual price of a fuel up or 
down to reflect a consumer’s bias for certain fuels. This 
subjective preference can be based on aesthetics, perceived 
safety, or relative ease of contracting. Since the net present 
return is the same for each unit of energy, regardless of the 
fuel chosen, a consumer can maximize his net present value 
by choosing the fuel that minimizes his net present cost. 
This is the economic criterion that is used in the Interfuel 
Competition Model. 

Although the above equations give the correct definition 
for the net present cost associated with a particular type of 
fuel, a more convenient form is used in the Interfuel Com¬ 
petition Model. This definition is recursive; the net present 
cost in one time period is defined in terms of the net present 
cost in the following period: 

NPQ=(1 + IN COME TAX RATE) VC t 

+ (1+INVEST. TAX CREDIT)/, + ( 6 ) 

Although Equation ( 6 ) is equivalent to Equation (4) above, 
this equation for the net present cost does not require a sum 
of variable costs and investments over time. However, to 


* This assumes that by switching from one fuel to another, a consumer does 
not face the risk of finding himself with an inadequate energy supply. The 
possibility of a general energy shortage and its impact on the decision making 
of energy consumers is discussed later. 
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calculate the net present cost in any time period with this 
equation, we must first calculate the net present cost in the 
following time period. In the Interfuel Competition Model 
the net present costs associated with each fuel and each 
time period are computed by working backwards through 
time, starting with the last year considered by the model 
and working backwards to the present. 


DECISION PATH OUTPUT FROM SWITCHING 
LOGIC 

The Interfuel Competition Model calculates the net pres¬ 
ent cost associated with switching from every possible fuel 
to every other possible fuel (or keeping the same fuel) in 
each year. The program then selects the fuel that minimizes 
the customer’s net present cost. Using this information, the 
model produces an array of decisions showing which fuels 
the customer should use over time for each possible initial 
fuel type. This array of decision for a hypothetical customer 
is represented graphically in Figure 2. 

If the consumer was initially using gas, the decision ar¬ 
rows in Figure 2 indicate that he should continue to use gas 
through the third year and then switch to oil. On the other 
hand, if he started with coal, he should continue to use his 
initial fuel for at least the first four years. Figure 2 also 
indicates that customer using gas in year 4 should switch to 
oil, even though a customer who had started with gas in the 
initial year would have switched to oil the third year. Thus, 
if the decisions shown in Figure 2 are followed, it would not 
be possible for this customer to be using gas in the fourth 
year. 


ILLUSTRATIVE EXAMPLES OF THE SWITCHING 
LOGIC 

The model’s switching logic can best be illustrated by 
simple examples. To keep the examples manageable, fuel 
choices are limited to gas (G), oil (O), coal (C), and dual- 
fired gas and oil (G/O). In the actual model additional fuel 
types are possible, including electricity, purchased steam, 
dual-fired gas and coal. In addition, we will ignore the dis¬ 
tinction between refurbished and remaining equipment in 
these examples. 



coal Q - - Q - *-Q-—O • • • O 

Figure 2—Decision paths—Switching logic output 


Example without fuel availability uncertainty 

The decisions that a customer with four possible fuel types 
must face in any given year are shown in Figure 3. At the 
end of year the customer could be using any one of the four 
possible fuels, as indicated by the four hexagons at the left 
of Figure 3. In fact, he may have several different types of 
equipment using different fuels. Some of his existing equip¬ 
ment will have to be retired, and he may need to add some 
new equipment to his inventory. For each type of equip¬ 
ment—new, currently using gas, currently using oil, cur¬ 
rently using coal, and currently dual-fired gas/oil—he must 
decide which fuel to use in the future. Decisions are shown 
in Figure 3 by small squares. Separate decisions are made 
for new equipment and for equipment currently using each 
of the possible fuels because the economics of switching 
fuels is different in each case. Once the customer has made 
all decisions for year I, the various pieces of energy con¬ 
suming equipment can be aggregated by fuel type. Thus, if 
the customer decides to continue using gas for his gas-fired 
equipment, and to purchase some new equipment that uses 
gas, the amount of gas-fired equipment carried over to the 
next year will include the total of both these equipment 
categories. In the following year, 1 + 1, the customer, after 
retiring some of his old equipment and purchasing some new 
equipment, is again faced with the same set of decisions. In 
each case the decisions are based on minimizing the net 
present cost associated with each possible fuel. 

Figure 4 illustrates how the consumer might make his 
decisions for new equipment, equipment currently using gas, 
and equipment that can use either gas or oil. It is assumed 
that the consumer starts in year 1-1 with ten units of ca¬ 
pacity in gas-fired equipment that can bum either gas or oil, 
as represented by the 10’s above the hexagons at the left of 
Figure 4. If four new units of capacity are purchased to 
replace two units of capacity for each type of equipment 
which must be retired, the three decisions for year I, rep¬ 
resented by the boxes at the left are: What is the best fuel 
type for: (1) the new equipment, (2) the eight units of ca¬ 
pacity currently using gas, and (3) the eight units of capacity 
currently using gas and oil? 

The Interfuel Competition Model treats equipment capa¬ 
ble of using more than one fuel (such as the gas/oil equip¬ 
ment in this example) as if it were a separate fuel type. After 
the model has allocated demand to dual-fuel equipment, a 
separate calculation determines the cheapest available fuel 
that the equipment can use in each year. Thus, gas and gas/ 
oil equipment may both use gas in a particular year. In this 
case the allocations to gas and gas/oil equipment must be 
summed to determine the total gas demand. However, if oil 
becomes less expensive than gas in the following year, the 
dual-fueled gas/oil equipment will automatically switch to 
oil without incurring any additional investment costs. Equip¬ 
ment based on gas alone will switch to oil only if the dis¬ 
counted cost savings are sufficient to outweigh the required 
investment.* 


* Multiple-fuel equipment may be required to use a minimum amount of each 
of its possible fuels. 
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Figure 3—Basic switching logic 
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ASSUME 


C 


OIL IS CHEAPEST FUEL. 

BEST DECISION IS TO REMAIN IN CURRENT FUEL AND START NEW EQUIPMENT IN GAS/OIL. 


G> 




CAPACITY = 10 GAS, 10 GAS/OIL 

_y v_ 

YEAR 1-1 


DEMAND = 8 GAS, 12 OIL 


CAPACITY = 8 GAS, 12 GAS/OIL CAPACITY = 6 GAS, 14 GAS/OIL 

, / DEMAND = 6 GAS, 14 OIL_ y 


YEAR I 


YEAR 1+1 


Figure 4—An example of the basic switching logic 


Let us assume that oil is the cheapest fuel and that the 
best decision at each point is to continue using the existing 
type of equipment or to purchase new equipment that is 
capable of burning both gas and oil. The consumer’s deci¬ 
sions and their effect on the capacity of his equipment and 
his demand for various fuels is shown in Figure 4. In year 
I—1 two of the initial ten units of capacity in equipment 
using gas only are retired, and the remaining eight units 
continue to burn gas only. Similarly, eight of the initial ten 
units of capacity in dual-fuel gas/oil equipment are main¬ 
tained through year I and combined with four new units of 
capacity capable of burning both fuels. Thus at the end of 
year I there are eight units of capacity in gas only and twelve 
units of capacity in gas/oil. Since the dual-fuel equipment is 
used with the cheapest fuel, oil, the fuel demand in year I 
is for eight units of gas and twelve units of oil. In year 1 + 1 
the process repeats itself, with the capacity of dual-fired 
equipment increasing from twelve to fourteen. 

Example with uncertain fuel availability 

To generalize the example to uncertain fuel availability 
we must differentiate between three general categories of 


unavailability. The first is operational reliability, this can be 
modeled by adjustments to O&M expenses to reflect inven¬ 
tory carrying costs and stockouts, as in any inventory prob¬ 
lem. The second is long-term shortages of specific fuel types 
caused by government regulations or scarcity; this uncer¬ 
tainty is treated explicitly in the model by introducing the 
probability of unavailability as illustrated in the example 
below. The third is a general energy shortage, where total 
supply is insufficient to meet demand. This is modeled by 
adding a term to the variable cost equal to the product of 
three terms: (1) the probability that a general energy short¬ 
age will occur in each future year, (2) the probability that a 
consumer will be forced to do without energy if he is using 
a particular fuel when an energy shortage occurs, and (3) 
the net cost to the consumer that results from doing without 
energy for the expected duration of the shortage. 

Figure 5 illustrates a simplified numerical example of the 
switching logic contained in the Interfuel Competition Model 
when fuel availability is uncertain. Only two types of equip¬ 
ment are considered in the example: that using gas only, 
and that capable of burning either gas or oil. For simplicity, 
it is assumed that there is no new, retired, or refurbished 
equipment; thus all of the existing equipment is carried over 
from one year to the next without additions, losses, or sep- 
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Figure 5—An example of switching logic with uncertain fuel availability 


arate decisions for refurbished equipment. It is also assumed 
that: (1) the best decision in each year is to remain with the 
current fuel unless that fuel becomes unavailable, (2) gas is 
the cheapest fuel, and (3) in a year when gas is unavailable, 
the duration of the shortage will force consumers to use oil 
for half their annual volume. In addition, it is assumed that 
there is a 20 percent chance that gas will be unavailable in 
each year. This is illustrated in Figure 5 by the probability 
node shown as a circle. 

The example starts at the end of year (I—1) with ten units 
of equipment capable of using gas only and ten units of 
equipment capable of using either gas or oil. In this case, 
the ten units of dual-fired equipment are run on gas since it 
is the cheapest fuel. If gas is available throughout the year 
I, none of the equipment is modified and twenty units of gas 
are demanded. However, if gas is unavailable in year I, the 
ten units of equipment on gas only are converted to allow 
them to burn either gas or oil. All the dual-fueled equipment 
is then operated on oil for half the annual volume, and on 
gas for the rest of the year. At the end of year I, either ten 
units of capacity will operate on gas and ten on gas/oil, or 
all of the equipment will have been converted to gas/oil, 
depending on whether or not gas was available during the 


year. The expected capacity at the end of year I is eight 
units of equipment capable of burning gas only, and twelve 
units of dual-fuel equipment. During year I there is an 80 
percent that the entire demand will be for gas, and a 20 
percent chance that the demand will be split evenly between 
gas and oil. Thus, the expected demand will be split evenly 
between gas and oil. The expected demand during the year 
is for 18 units of gas and two units of oil. 

In the year 1+1, the process repeats itself. The expected 
capacity of equipment capable of burning only gas drops 
from 8 to 6.4, while the expected capacity of dual-fuel equip¬ 
ment rises from 12 to 13.6. However, there is still an 80 
percent chance that the entire demand during year 1+1 is 
for 18 units of gas and two units of oil, unchanged from the 
previous year. This example illustrates the fact that the 
possibility of fuel shortages may cause consumers to protect 
themselves by switching to dual-fired equipment without 
changing the expected demand for individual fuels. This type 
of consumer behavior, which is incorporated into the Inter¬ 
fuel Competition Model, leads to volatile energy markets 
marked by large fractions of total energy demand that switch 
rapidly from one fuel to another. 
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OTHER ISSUES 

Fuel availability is just one of many issues considered in 
designing and implementing the Interfuel Competition 
Model. Other phenomena of importance include: 

• Lags in consumer response to energy market changes 

• Different planning horizons used by energy consumers 

• Legal and physical constraints that limit a consumer’s 
ability to switch from one fuel to another 

• Rate structures employed by electric utilities 

• Effects of conservation, regulation, and growth within 
the gross national product on energy demand 

• Interdependence of prices and volumes of all fuels. 


CONCLUSION 

This model is among the first models to predict market 
demand and market share by examining the behavior of 
specific customer groups. Most other demand forecasting 
methodologies rely on price elasticities or market share 
curves, and, consequently, they predict a smooth response 
of demand to price differentials between fuel types. Of 
course, price response depends on specific data used in the 
model, but typical demand curves derived from the model 
are not smooth. Gas demand forecasts from the model based 
on dynamic programming, probabilistic decision theory, and 
detailed market segmentation should be more accurate than 
forecasts from traditional models. 




General equilibrium models for energy policy analysis 
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INTRODUCTION 

In this paper we shall provide an elementary overview of 
the concept of general economic equilibrium in the concrete 
context of the U.S. energy market.* With this aim in mind, 
we shall introduce and discuss several important, large-scale 
energy models which embody the assumptions of the meth¬ 
odology of economic equilibrium. The models which will be 
discussed were all developed since the OPEC embargo of 
1973—an event which has provided the impetus for devel¬ 
opment of sophisticated computer models useful in energy 
planning and policy-making. 

In the first section, the models of Hudson and Jorgenson 1 
and Hnyilicza 2 are introduced. The discussion here will 
hopefully provide the reader with a clear diagrammatic ex¬ 
position of the concept of general economic equilibrium. By 
“economic equilibrium” we mean a state of the economy in 
which prices are such as to equate the supply and demand 
for all commodities under consideration. These two partic¬ 
ular models provide a good starting point for our discussion 
since they are very broad in scope, encompassing both the 
energy and the non-energy sectors of the U.S. economy. In 
doing so they take account of the important linkages which 
exist between the energy and non-energy sectors. 

Then in the second section, we introduce the SRI National 
energy model, sometimes referred to as the SRI-Gulf model 
since it was originally developed with the collaboration of 
the Gulf Oil Corporation. Our focus in this section is on the 
algorithm through which economic equilibrium is deter¬ 
mined in a dynamic economic context. An understanding of 
the algorithm not only sponsors an in-depth understanding 
of the SRI model and “how it works,” but permits an ap¬ 
preciation of the important role of computation in large- 
scale models. Finally, in the last section, we conclude with 
some remarks on the role of economic models in energy 
policy analysis. 


* The research discussed in this paper was sponsored by the Office of Policy 
Research and Analysis of the U.S. National Science Foundation. The report 3 
on which this paper is based was co-authored by the present author, and by 
Dr. Dale M. Nesbitt of Decision Focus, Inc. of Palo Alto, California. 


THE HUDSON-JORGENSON/HNYILICZA MODELS— 

AGGREGATE MODELS OF ENERGY AND 

ECONOMIC GROWTH 

Background 

In 1974, Edward Hudson and Dale Jorgenson 1 introduced 
a model of energy and economic growth. The Hudson-Jor¬ 
genson model consisted of two somewhat separate submod¬ 
els: a macro-economic model of economic growth, and a 
multi-sector interindustry model. The model was the first to 
make use of a new and advanced methodology for econo¬ 
metric estimation in a general economic equilibrium context. 
This methodology is known as “translog economic model¬ 
ing” in the technical literature. 

An ostensible problem with the Hudson-Jorgenson model 
was its failure to integrate its two submodels in a completely 
satisfactory way. It was not clear that the inputs and outputs 
of the two submodels were compatible. This difficulty pro¬ 
vided the motivation for Esteban Hnyilicza 2 to develop a 
model which, while very similar to the Hudson-Jorgenson 
model, differed from it in its treatment of economic growth. 
Specifically, Hnyilicza has attempted to build a model in 
which the multi-sector inter-industry model and the macro- 
economic growth model are satisfactorily integrated. The 
Hnyilicza model also differs from the Hudson-Jorgenson 
model in being much more highly aggregated. Specifically, 
it consists of only two sectors, namely the energy and non¬ 
energy sectors of the economy. In other important respects, 
such as use of the “translog” methodology, the two models 
are much the same. 

In this section, we are going to discuss the Hnyilicza 
model at some length. The purpose of discussing this model 
rather than the Hudson-Jorgenson model is that our dia¬ 
grammatic mode of exposition of the methodology used in 
both models requires that we limit ourselves to a simple 
model with very few sectors. 

Introduction 

Hnyilicza has constructed an integrated model for the 
purpose of analyzing the relationship between economic 
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growth, and the (equilibrium) prices and quantities of both 
energy and non-energy goods. His model provides the basis 
for computing a trajectory of future market clearing (i.e., 
“equilibrium”) prices and quantities of the following com¬ 
modities: 

1. Energy consumption goods 

2. Non-energy consumption goods 

3. Energy intermediate goods 

4. Non-energy intermediate goods 

5. Capital services to the energy sector 

6 . Capital services to the non-energy sector 

7. Labor services 

8 . Energy imports 

9. Non-energy imports 

These outputs are generated on a period-by-period basis as 
the solution to a model consisting of fifty-five (nonlinear) 
simultaneous equations. By analyzing time-series projec¬ 
tions of the prices and quantities of these goods, Hnyilicza 
can diagnose the relationship between economic growth on 
the one hand, and the relative prices of energy and non¬ 
energy goods on the other hand. It is also possible to study 
the rate of substitution of capital and labor for energy, as 
well as the economic impact of alternative tax and conser¬ 
vation policies. 

Overview of the model 

Figure 1 displays the overall structure of the Hnyilicza 
model. There are three basic model sectors: industrial pro¬ 
duction, consumption, and investment.* The fact that these 
sectors are enclosed by solid as opposed to dashed lines is 
supposed to describe the fact that the outputs of these sec¬ 
tors are determined endogenously within the model. This is 
not the case with the government sector and the foreign 
sector. The latter two (which are enclosed by dashed lines) 
are largely exogenous to the model.* 

Before delving into the details of the three sectors, let us 
briefly summarize what takes place in the model in hopes of 
better motivating Figure 1 and in hopes of providing the 
reader with a guide to the model. One preliminary word of 
warning is in order. It is extremely difficult to describe what 
“goes on” in any general equilibrium economic model on a 
sector-by-sector basis. The reason for this is simply that 
what happens in any one sector affects every other sector. 
Nonetheless, we have tried to motivate the model by means 
of a sector-by-sector discussion since there seems to be no 
satisfactory expository alternative. 

Two things happen in the consumption sector. First, the 


* It is perhaps a bit artificial and unnecessary to identify an “investment 
sector” which is distinct from the consumption and production sectors. We 
have only done so in order to facilitate a description of what goes on in a 
“growth” model. 

t Specifically, the behavior of the government sector is almost completely 
exogenous. In the import markets, whereas the supply curve for imported 
goods is exogenous, the demand curve for imports (energy and non-energy) 
is endogenous. 


“national household’s” inter-temporal utility function de¬ 
termines the split between present consumption and future 
consumption (i.e., savings). More specifically, the house¬ 
hold is assumed to determine a utility-maximizing split be¬ 
tween consumption and savings as an (econometrically es¬ 
timated) function of wealth, the wage rate, and government 
transfer payment levels. Second, the consumer decides upon 
an intra -period split between leisure, energy consumption 
goods, non-energy consumption goods, and capital services. 
The optimal split is that which is utility-maximizing, subject 
to the consumer’s budget constraint. This split will of course 
be a function of the relative prices of the various consump¬ 
tion goods, and the wage rate, among other things. 

The two industrial production sectors—energy and non¬ 
energy—determine an optimal (profit-maximizing) level of 
outputs, and an optimal (profit-maximizing) configuration of 
inputs used in producing the equilibrium output levels. 

Finally, the investment sector determines the level of cap¬ 
ital services which will be available in any given period. As 
we shall see, this level depends on both the efficiency of 
capital, and on the level of capital stocks in the preceding 
period. The investment sector also determines the overall 
level of gross investment in any given period, as well as the 
fractions of gross investment going to the energy sector, the 
non-energy sector and the household. 

A diagrammatic exposition 

An important question arises concerning the best way in 
which to explain and motivate a model which is as general 
as the Hnyilicza model. Two approaches seem to be 
possible: a mathematical approach and a diagrammatic ap¬ 
proach. We have settled upon a diagrammatic exposition of 
the model. More specifically, we shall motivate the model 
by discussing each of the three sectors (production, con¬ 
sumption, and investment) from the standpoint of the de¬ 
mand curves and the supply curves which are implicit in the 
equations which characterize economic behavior in each 
sector. This strikes us as a compact and intelligible mode of 
exposition. Furthermore it permits us to conclude our dis¬ 
cussion with what we hope to be a highly appealing summary 
of the model: namely, a picture (Figure 5) in which we 
superimpose the various supply and demand curves which 
we “extract” from each of the three sectors. The price- 
quantity pairs of all of the points of intersection of these 
supply and demand curves will clearly constitute a general 
economic equilibrium. And these equilibrium prices and 
quantities coincide with the values of the endogenous vari¬ 
ables which are determined when the model is solved. 

In light of our decision to explicate the model in terms of 
the supply and demand relationships that are implicit in the 
equilibrium equations of the model, a caveat must be issued 
from the outset. Whenever the reader sees a picture of a 
supply curve or a demand curve, he should realize that he 
is beholding an illusion. For in point of fact, there are no 
such things as supply and demand curves per se. There are 
only conditional supply and demand curves. Let us illustrate 
this point since it often engenders confusion. Consider a two 
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Figure 1—The Hnyilicza model structure 


commodity world consisting of coffee and sugar. In general, 
the reader would be hard pressed to spell out his price- 
demand relationship for coffee. He would only be able to 
do so if he knew how much sugar he would be allotted. Thus 
we can speak of a demand curve for coffee conditional upon 
a particular consumption level of sugar. For this reason, it 
is somewhat artificial to characterize market equilibrium 
below in terms of a set of supply curves and demand curves.* 

Methodology of the model 

Industrial production sector 

The function of the production sector in the context of the 
larger model is twofold: 

a. to determine an equilibrium factor input mix which is 
optimal (profit maximizing) for each industry in pro¬ 
ducing its total output; 

b. to determine the total (equilibrium) output levels that 
meet the demands of the intermediate market and of 
the final goods market. 

As we saw in Figure 1 and as we see in more detail in 
Figure 2 on the next page, we can think of the production 
sector as an input-output structure. We summarize this 
input-output structure in Table I. 


t Nonetheless, it will be true that if we somehow knew the equilibrium 
quantities of all commodities, then we could sketch a set of conditional supply 
and demand curves whose points of intersection do constitute the market 
equilibrium. The problem of course is that one does not know the equilibrium 
in advance. 


It is important to realize that even though we are speaking 
of production in “input-output” terms, the Hnyilicza model 
is not an input-output model in the classical sense of that 
term. Rather, it is a generalized input-output model. In such 
models the input-output (transactions) coefficients, rather 
than being given exogenously (as would be the case in the 
economy whose technologies permitted no substitutability 
among inputs), are functions of the prices of all inputs and 
the configuration of final demand. Specifically, we can write 
% =f(w,p,y,x) 

where: w is the vector of input factor prices 
p is the vector of output prices 
y is the vector of final demands 
x is the vector of gross outputs. 

The equilibrium coefficients {«„*} that are implicitly 
solved for by the model designate the profit-maximizing 


TABLE I 

Inputs 



Labor Services 

Capital Services* 

Imports 

Energy Intermediate Goods 

Non-Energy Intermediate Goods 


Energy Industry Outputs 


Non-Energy Industry Outputs 

Energy Consumer Goods 


Non-Energy Consumer Goods 

Energy Intermediate Goods 


Non-Energy Intermediate Goods 
Investment Goods 


‘Capital services are the annual services generated by existing stocks 
of capital goods. These services are provided to the productive process. 
"Capital goods" have a life exceeding one year, by definition, whereas 
"intermediate goods" are consumed by the production process within a year. 
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number of units of commodity i which are utilized in the 
production of each unit of commodity j. Let us now press 
towards a deeper understanding of what is going on in the 
production sector. A result of this will be an enhanced un¬ 
derstanding of the equilibrium coefficients {a i} * }. 

The most straightforward way in which to characterize 
what goes on in the production sector of the model is to 
depict the various supply curves and demand curves which 
are implicit in the equilibrium equations which characterize 
rational economic behavior on the part of the two industries: 
energy and non-energy. 

The entrepreneur must make essentially two different 
types of decisions. He must decide upon a profit maximizing 
level of output of the commodities he manufactures; and he 
must decide upon a profit maximizing mix of inputs to be 
used in meeting his (optimal) production schedule. 

Clearly his optimal choice of output levels and input mixes 
will be a function of the relative prices of outputs and inputs. 
Indeed, it is customary to speak of the relationship between 
the price of a given input and the quantity demanded by the 
entrepreneur at that price as the derived factor demand 
curve of the input (or “factor”) in question. And the rela¬ 
tionship between the optimal output level of a given output 
and the price of the output is summarized by the supply 
curve of the output. 


With this in mind the reader is referred to Figure 2, where 
we identify the various derived demand curves and supply 
curves that are generated within the production sector. Let 
us summarize the information contained in the figure. 

The derived factor demand curves and the supply curves 

There is one derived factor demand curve for each input. 
Since both the energy and the non-energy sector use five 
inputs, there are therefore ten derived demand curves in all: 

Energy Non-Energy 

Labor Services 
Capital Services 
Energy Intermediate 
Goods 
Non-Energy 
Intermediate Goods 
Imports 

The derived demand curves associated with these input de¬ 
mands are illustrated in Figure 2. 

With respect to supply, the energy sector produces only 
two goods whereas the non-energy sector produces three 


Demands: 


Labor Services 
Capital Services 
Energy Intermediate 
Goods 
Non-Energy 
Intermediate 
Goods 
Imports 
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goods: 

Energy Non-Energy 

Energy Non-Energy 

Consumption Consumption 

Goods Goods 

Supply: Energy Intermediate Non-Energy 

Goods Intermediate 

Goods 

Investment Goods 

The associated supply curves are indicated in Figure 2. 

An input-output interpretation 

We are now in a position to give an interpretation to the 
equilibrium Input-Output coefficients {fly*}. Note in Figure 
2 that in two (and only in two) of the markets pictured there 
do we have well-defined supply and demand curves. This is 
the case in the markets for energy intermediate goods and 
non-energy intermediate goods. Suppose now that we could 
“complete” our description of the remaining eight markets 
by obtaining the missing supply and demand curves. (As 
will be seen, these missing curves are generated in the other 
two sectors of the model to which we shall turn shortly.) 
Suppose finally that we were to take note of the equilibrium 


point in these ten markets, that is, the equilibrium prices 
and quantities of the ten commodities. 

It would then be straightforward to compute the equilib¬ 
rium input-output coefficients either in dollar terms or in 
physical terms—using simple arithmetic. The collection of 
coefficients so obtained would coincide with the collection 
{fly*}. That is, these are the equilibrium input-output coef¬ 
ficients associated with profit-maximizing behavior on the 
part of the industries with respect to the choice of input mix 
and the choice of output level. 

The consumption sector 

Figure 3 portrays the consumption sector in some level of 
detail. Looking at the top part of the figure it is clear that 
two things take place in the consumption sector model. 
First, an “intertemporal” utility function determines a util¬ 
ity-maximizing split between present and future consump¬ 
tion. This utility function is an econometrically estimated 
function of: 

Time 

Interest rate 

Existing wealth 

Wage rate 

Government payments to household (e.g., welfare) 



Figure 3—The household sector 
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Second, an “intratemporal” utility function determines 
the split within a given period of household expenditures 
between the four consumption goods, namely energy con¬ 
sumption goods, non-energy consumption goods, capital 
services, and leisure. By capital services we mean the an¬ 
nualized value of the services which flow from consumer 
durables or residential stock. The intratemporal utility func¬ 
tion which performs this split is an econometrically esti¬ 
mated function of price, the level of consumption of each 
good, and time. 

A supply and demand curve interpretation 

A deeper understanding of the consumption sector can be 
obtained by considering the supply and demand curves 
which are implicit in the equilibrium equations characteriz¬ 
ing utility-maximizing behavior on the part of the household. 
These price-quantity relations are shown in the lower por¬ 
tion of Figure 3. Interpretation of all of these curves is 
straightforward with the exception of the supply curve of 
labor services. This curve is in fact the inverse of the de¬ 
mand curve for leisure services. As the wage rate increases, 
leisure becomes more expensive. Work is substituted for 
leisure, which increases the supply of labor. 

Investment and capital accumulation 

Formally speaking, the investment sector is part of the 
production sector. Nonetheless, it is useful to isolate it as 


a separate sector for the purposes of enhancing an under¬ 
standing of capital accumulation in the Hnyilicza model. 
The situation is pictured in Figure 4. Consider the upper 
part of the figure. The supply of new investment goods that 
come on stream in a given year t appears on the left. Next, 
the supply of new investment goods is split three ways into 
the shares of investment goods going to the energy, non¬ 
energy, and domestic sectors. This split is realized by the 
use of a multiplier which reflects historical ratios in the 
apportionment of investment goods. 

Moving further to the right, we add in the depreciated 
value of capital stock for the respective sectors. These are 
the values of stocks at the end of the previous year. The 
result of this addition is the current level of capital stock. 

Moving yet further to the right in Figure 4, the values of 
the current level of capital stock in the two industrial sectors 
are multiplied by the efficiency of capital. The result of this 
operation is defined to be the level of capital services in 
period t. 

We exhibit the current supply of capital services generated 
in this fashion within each of the two industries as two 
supply curves. The reader will note that these supply curves 
are perfectly inelastic. The reason for this is that the level 
of capital services is not a function of any current prices. 
One might think that this level would be a function of the 
current price of new investment goods. This is not the case, 
for in this model there is a one period lag between the time 
when capital goods are produced and the time when they 
begin to generate capital services. The investment goods 
which come “on stream” in year k and which appear in the 
upper left of Figure 4 were actually produced in year (& -1). 
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Two other price-quantity curves appear in the figure. 
First, there is a demand curve for investments goods. This 
demand curve is seen to be perfectly elastic. Moreover, it 
appears as a dashed line. Why? We have used a dashed line 
to indicate that this demand curve is an artificial construct. 
For in the Hnyilicza 2 model, the (equilibrium) level of in¬ 
vestment goods is, in fact, “supply-determined.” Whatever 
quantity of investments goods is supplied is assumed to be 
absorbed—irrespective of own price. It is for this latter 
reason that the curve appears as perfectly elastic. 

In point of fact, it is possible to introduce a demand 
function for investment goods in the Hnyilicza model. For 
one of the basic equilibrium conditions of the model is that 
investment equals income minus consumption. [This equi¬ 
librium condition is imposed on the model.] Consumption, 
in turn, is a function of income and of the price of labor, 
capital, etc. Thus, the demand for investment goods is in¬ 
deed a function of “prices” in the Hnyilicza model. Our 
point above is simply that it is not helpful to characterize a 
market of investment goods per se, and to plot the demand 
for investment goods as a function of the price for invest¬ 
ment goods. For, as we said above, whatever quantity of 
investment goods is produced is assumed to be demanded. 

Last of all, there is a demand curve for savings. This 
appears as a dashed vertical line. Why? The reason for this 
is that there is no “demand curve” for savings per se in the 
model. Rather, savings is assumed to equal investment. 
Once again, just as in the case of the demand for investment 
goods above, it is possible to assert that there is a demand 
function for savings embedded in the Hnyilicza model. For 
the so-called “static equilibrium condition” of this type of 
growth model is that savings equals income minus “desired” 
consumption. This relationship is imposed on the model, 
and serves to relate industry demand for savings to con¬ 
sumption and hence to relative prices, since consumption 
depends in part on prices. 

The reader who finds the rather arbitrary treatment of 
investment and savings perplexing should be aware of the 
motivation for the static equilibrium conditions such as 
“Savings=Investment” that are imposed on the model. The 
rationale for these assumptions is simply to insure consis¬ 
tency between the aggregate savings decisions by individuals 
on the one hand, and the aggregate investment decisions of 
producers on the other hand. There would be no need, in 
principle, for imposing static equilibrium conditions if the 
model included a full set of futures markets, and if an equi¬ 
librium set of prices and quantities across time were com¬ 
puted simultaneously, as is the case in the SRI model. Of 
course, to point this out is not to make a value judgment 
about the quality of the SRI model versus the Hnyilicza 
model, since the emphases of the two models are different. 

General equilibrium in the Hnyilicza model 

Figure 5 shows the result of collecting together the various 
supply and demand curves that we have extracted from the 
production sector, the consumption sector, and the invest¬ 
ment sector. The point of intersection of all of these curves 


coincides with the equilibrium prices and quantities that 
Hnyilicza’s fifty-five simultaneous equations are solved for. 
We can thus interpret equilibrium in terms of supply-demand 
balance in the following markets: 

1. Capital Services to the Energy Industry 

2. Capital Services to the Non-energy Industry 

3. Labor Services 

4. Energy Imports 

5. Non-energy Imports 

6. Energy Consumption Goods 

7. Non-energy Consumption Goods 

8. Energy Intermediate Goods 

9. Non-energy Intermediate Goods 

10. [Investment Goods] 

11. [Savings] 

Time-series projections of these equilibrium prices and 
quantities are the principal outputs of the Hnyilicza model. 
The following variables are exogenous to the model and are 
required as inputs: 

• Inventory investment 

• Labor force 

• Tax rates 

• Government expenditures 

• Exports 

• International financial claims 

• Government transfer payments (e.g., welfare) 

• Rate of unemployment 

• Import prices 

• Rate of replacement of capital stock 

• Initial year private national wealth 

• Initial year capital stock 

• Quality of capital (a constant, multiplied by capital 
stock to give capital services) 

A note on the new version of the Hudson-Jorgenson 
model* 

Hudson and Jorgenson have recently reworked their orig¬ 
inal model. Their new model is very similar to the Hnyilicza 
model, especially as regards the use of the variable coeffi¬ 
cient input-output methodology in an economic equilibrium 
context. Where the two models differ is in the degree of 
disaggregation of the production sector. As we saw, the 
Hnyilicza model’s production submodel consisted of two 
sectors: energy and non-energy. The Hudson-Jorgenson 
production submodel consists of six sectors: 

• Coal 

• Crude petroleum 

• Refined petroleum products 

• Electricity 

• Natural gas (crude) 

• Delivered gas 


* The author is grateful to Dr. Edward Hudson for a helpful discussion. 
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THE SRI NATIONAL ENERGY MODEL 


Introduction 

The SRI National Energy Model is an economic equilib¬ 
rium model of the U.S. energy system covering all major 
fuels from primary resources to end use energy consumption 
(i.e., energy service demands) over the period 1977 to 2025. 
The model is regionally disaggregated with nine demand 
regions (such as New England), twenty primary resource 
supply regions (such as Alaska North Slope), and 600 trans¬ 
portation and distribution links. Conversion technologies 
such as coal gasification, crude oil refining, and electric 
power generation by type of fuel are explicitly modeled. The 
model computes regional prices and quantities of all energy 
forms, as well as energy technology requirements over the 
period of 1977 to 2025 by balancing supply and demand. 

The SRI model differs in several important respects from 
the models considered in the previous section. To begin 
with, the Hudson-Jorgenson and Hnyilicza models are 
highly aggregated models of the overall economy. In con¬ 


trast to this, the SRI model is a very disaggregated and 
detailed model of the energy sector of the U.S. economy. 
All this detail can be represented graphically by means of a 
“spatial network.” We shall discuss this network just below. 
Above and beyond this, the SRI model makes use of a 
different methodology. While the model is an economic 
equilibrium model in the sense that it determines prices and 
quantities which are in supply-demand balance,—just as the 
other models do—it does not make use of the “translog” 
methodology. There are no variable input-output coeffi¬ 
cients. Rather, economic equilibrium is characterized in a 
much more direct manner. One solves directly for prices 
and quantities which are in balance. Moreover, we can vi¬ 
sualize “equilibrium” in this model in terms of a supply- 
demand balance at each point in the spatial network referred 
to above. 

Partly for this reason, we shall start off by discussing the 
spatial energy network which lies at the heart of the model. 
Next we shall describe what is meant by “equilibrium” in 
this context. Finally, we shall gain a deep understanding of 
the model by analyzing the algorithm used to achieve a 
supply-demand balance throughout the network. 
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A spatial network representation of the energy market 

The essence of the SRI model is a spatial network rep¬ 
resentation of the energy system. In the network, the major 
elements of the U.S. energy system—from the primary re¬ 
source supplies through the end use energy demands—are 
described in relationship with each other. A sample energy 
network is shown in Figure 6. In this figure, the resource 
supply curves are at the bottom; the usable energy demand 
curves are at the top. Between these curves is the network 
describing the entire energy system. The current network 
has about 2400 materials, processes, and transportation 
links. A material is a primary resource product, or usable 
energy form at a specific location. A material is represented 
by a node in Figure 6. A process represents a sector of the 
energy industry such as coal mining or gasification at a 
specific location or a class of consumers using a particular 
energy-consuming device. A transportation link represents 
the process of moving a material from one location to an¬ 
other. 

To get a sense of the many paths in the network, consider 


first the path where coal is mined, converted into synthetic 
(high Btu) gas, piped to a demand center in a demand region, 
distributed to industrial users, and consumed as boiler fuel 
to produce steam. The same end use market could be sup¬ 
plied by coal transported by unit train, distributed to the 
same industrial users, and used in a coal boiler to produce 
steam. These two paths can be traced in Figure 6. In the 
SRI National Energy Model, there are 24 end uses (such as 
industrial steam) in each of the nine demand regions, and 
thirty primary resource supplies (such as coal) in the various 
resource basins. The alternative conversion technologies in 
the model include all important types of electric power gen¬ 
eration (in base, intermediate, and peak load applications), 
crude oil refining, coal gasification, methanol from coal, 
hydrogen production from coal, transportation, distribution, 
and end use conversion. 

The model explicitly considers the evolution of the energy 
system through time—it calculates prices and quantities in 
a number of time periods. It is important to recognize that 
the solution to this “dynamic” model is not equivalent to 
using a “static” model in each of the time periods. Rather, 
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Figure 6—Example of elements of the energy model 
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the prices and quantities in each period are described by 
equations interrelating both past and future prices and quan¬ 
tities. 



DEMAND 

FUNCTION 

© 


On the meaning of economic equilibrium in the SRI model 

In their earlier research, Brock and Nesbitt 3 have drawn 
upon economic theory to establish the following results con¬ 
cerning economic equilibrium in spatial networks: 

1. There exists an aggregate demand curve and an ag¬ 
gregate supply curve at every material node (circle) in 
the network; 

2. There exists a supply curve and a demand curve on 
every link in the network. Any such pair characterizes 
the supply/demand conditions governing the behavior 
of the (unique) conversion or transportation process 
attached to the link. 


SUPPLY FUNCTION /T\ 
TRANSFORMATION W 



© DEMAND 
FUNCTION 
TRANSFORMATION 


Figure 7—SRI model algorithm 


The equilibrium prices and quantities are those for which 
all aggregate supply/demand curve pairs are simultaneously 
balanced at all material nodes; or equivalently for which all 
supply/demand curve pairs are balanced on all the links of 
the network. 

The operation of the model 


We will now discuss each of these four steps in some detail, 
and in a less abstract manner. 

1. Inverse Supply Function: In the first step of the algo¬ 
rithm, the inverse supply function describing each of the 
primary resources at the bottom of the network in Figure 6 
is applied. The algorithm begins with a “guess” at the pro- 


The SRI model is best described in terms of the algorithm 
used to solve for equilibrium, and in terms of the relationship 
of the algorithm to the spatial network and its components. 
Therefore, in this section we shall describe the algorithm in 
considerable detail. When we say that this algorithm 
“solves” the spatial network we simply mean that it deter¬ 
mines the set of equilibrium prices and quantities which 
obtain at each point (link and node) in the network. 

The algorithm consists of four essential steps which are 
repeated at each iteration. The procedure is illustrated at a 
fairly abstract level in Figure 7. Beginning with an initial 
“guess” at the equilibrium quantity q k - x , the four steps of 
the algorithm are: 

1. To apply (inverse) resource supply function* 

Pk=S i ] 

2. To proliferate prices up the network (supply function 
transformation) 

Pk+i=T s [p k ] 

3. To apply usable energy demand function 

Qk+i=D[p k+1 ] 

4. To proliferate quantities down the network (demand 
function transformation) 



An inverse supply function expresses price as a function of quantity. 


Figure 8—Illustrative gas network 
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duction level over time for each of the primary resources 
(denoted q k -ft ) as in Figure 7, where the t denotes a vector 
over time). 

Using the guess q k - t {t), the inverse supply function is 
applied to give p k (t), the price of the corresponding primary 
resource. Schematically, Step 1 is given by 

The inverse supply functions are given exogenously, one for 
each primary resource. 

The first step of the algorithm is to guess initial quantities 
q A °, q B ° of natural gas and synthetic gas respectively and, 
using the natural gas and synthetic gas supply curves, to 
calculate the corresponding prices p< A 1 and p B l for natural 
and synthetic gas. These prices apply at the process boxes 
A and B in Figure 8. 

2. Supply Function Transformation: In the second step of 
the algorithm, the intermediate technology submodels are 
used to compute product prices given input fuel prices. 

The technology models begin by assuming that the mea¬ 
sure of profitability to a firm which builds a unit of capacity 
at time t is given by the net present value of the cash flow 
generated by that unit of capacity: 


t+L 


NPV(t)=-c(t )+£ 

T=t 


p(T)~4>(D 

(1 +r) T ~ t 


where 


c(t) =capital cost of the new unit of capacity built at 
time t; 

p(7)=product price at time T; 

<MT)=operating cost (including fuel cost) at time T; 
/“discount rate (cost of capital) 

L=life of technology. 


In order to simplify our discussion, we will assume that 
technologies have a useful life of three years (L=3) and that 
we wish to consider the interrelationship of prices over a 
three-year time horizon. 

It is known from elementary economic theory that at 
equilibrium, profit maximizing firms earn zero profit. Ex¬ 
pressed in terms of net present value, the zero profit as¬ 
sumption is equivalent to: 

NPV(t)=0 t= 1,2,. . . 


where we have assumed that the cost of debt and equity 
capital are included in the discount rate r. Assuming 
NPV{t )=0 for new plants built in Years 1, 2, and 3, we can 
write: 


c(i)=p(i)-<MD+ 
c(2)=p(2)—<H2)+ 
c(3)=p(3)—<M3)+ 


1+r 

(1+r) 2 

Pi 3)-<M3) , 

pi4)-fi4) 

1 + r + 

(1+r) 2 

pi4)-fi4) ( 

pi5)-fi5) 


1 +r 


(1 - r ) 2 


which is trying to decide whether to add new capacity at 
time t =3 knew with certainty what future prices [p(4) and 
p(5)] and future operating costs [<f>(4) and f(5)] would be, 
it could, by rearranging the third equation above, calculate 
the minimum price in Year 3, p( 3), below which it would 
not add capacity and above which it would add more ca¬ 
pacity.* 


p(3)=<M3)+c(3)- 


p(4)— fj4) 
1 + r 


P( 5)~<H5) 
(1+r) 2 


= <M3) + capital charge 

Knowing p( 3), </>(3), pif), and f(4), the second equation of 
the above three can be used to solve for pi 2). A similar 
calculation gives p( 1). 

To summarize, for each technology, given the fuel prices 
(contained in fit)), given assumptions about prices and op¬ 
erating costs past the end of the model horizon, and given 
process economic data (c(r) and fit)), product prices are 
calculated recursively backward in time 

Pi 3), P(2), pi 1) 


as indicated above. When we arrive at the top of the net¬ 
work, we will have calculated the prices p k+1 it) in Figure 7. 
We have denoted this upward proliferation of prices by using 
the notation P k+1 it)=T s [p k it)]. 

Returning to Figure 8, Step 2 of the algorithm involves 
taking the prices p A °, p B ° (and the quantities q A °, q B °) and 
determining the corresponding prices pf, p 2 ° at the top of 
the network. The first step is to compute the average price 
of gas entering the pipeline at a in Figure 8 as follows: 

- PaV+Pb^b 0 
p <+< 

The price of gas delivered to the two end users (i.e., to /8 in 
the figure) and would simply be 

Pi°=P2°= f+c. 


3. Demand Function: The third step in the algorithm is to 
apply the direct demand function at the top of the network 
using the prices p k+1 it) computed on the way up in Step 2. 
That is, an estimate of demand is obtained by applying the 
demand function 

q k+1 it)=D[p k+1 it)]. 

Returning to Figure 8, this step of the algorithm involves 
taking the price pf of residential gas determined by Step 2 
and finding the residential demand ft 0 at that price using the 
residential demand curve. A similar procedure yields the 
industrial demand qf corresponding to the industrial gas 
price pf determined in Step 2. 

4. Demand Function Transformation: In the fourth step 
of the algorithm, the technology submodels are used to com¬ 
pute fuel requirements given product demands, and market 
share submodels are used to divide demand among compet- 


which interrelates the equilibrium prices (and, of course, the 
capital and operating costs and the discount rate). If the firm 


* Equilibrium would obtain at the price p(3) itself because the firm would just 
be indifferent between adding and not adding capacity. 
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ing technologies based on the price computed for those 
technologies in Step 3. We will discuss both calculations 
briefly. 

First, each technology is characterized by a thermal effi¬ 
ciency e(t ) for new units built at time t and thus by an 
average efficiency e(t ) for all plants in place at time t. Hence, 
the fuel required is given by 

Second, if the quantity q k+2 (t) is to be split among a number 
of competing suppliers, a “market share” model is applied 
to the quantity q k+1 {t) to split it among competing supply 
technologies based on the prices computed for those tech¬ 
nologies in Step 2. We do not have space enough to discuss 
this market split model here. 

This step of the algorithm is better explained in terms of 
the network in Figure 8. Referring to that figure, the fourth 
step of the algorithm actually consists of two operations. 
First, the residential and industrial quantities q t ° and q 2 ° 
respectively determined in Step 3 are added to give the total 
demand for gas q=q/ ) +q 2 () at (3 in the figure. In order for 
the pipeline to deliver q units of gas at (3, it must take in 
q/e units of gas at a, where e is the thermal efficiency of 
the pipeline, including transportation losses. 

Given the demand q/e of gas at a, we then apply the 
market share model to compute how much of that gas would 
be synthetic and how much would be natural at the prices 
p A °, Pb° determined in Step 2. That is, we would compute 

Qa 1 = MS a (p a 0 , p B °)xq/e 

q B 1= MS B (p A °, p B °)xq/e 

where MS A (p A °, p B °) is the fraction of the demand that would 
be satisfied by Process A (natural gas) if the prices of natural 
and synthetic gas were p A and p B ° respectively, and simi¬ 
larly for Process B. 


Characterization of equilibrium in the network 

If <?/=« 4 ° and q B 1 —qB° equilibrium has been achieved 
(recall that q A ° and q B ° were our initial guesses in Step 1). 
If not, we return to Step 1 with the new guesses q A l and q 
and repeat the four steps, as many times as necessary to 
achieve equilibrium. 


THE ROLE OF ECONOMIC MODELS IN 
NORMATIVE POLICY ANALYSIS 

We shall conclude the present essay by issuing some pre¬ 
cautionary remarks about the role of economic models in 
normative public policy analyses. The interested reader will 


find an elaboration of these remarks in Chapter I of Brock 
and Nesbitt. 3 

It must be appreciated that the models discussed in this 
report are economic in nature. That is, their aim is to project 
prices and quantities of energy (and, sometimes, non-en¬ 
ergy) commodities. Yet normative policy analysis is inter¬ 
ested not only in the economic dimension of alternative 
policies, but also in the political and the ethical dimensions. 

For example, a policy maker might well wish to have at 
least probabilistic information about the likely price/quantity 
impacts of pursuing alternative strategies. For an example 
of this, see the companion paper presented by Dr. Steven 
N. Tani. Tani points out how price quantity data generated 
by an energy model can be used to make economic 
welfare judgments about alternative policies by means of the 
criterion of “net social surplus.” This is important—indeed, 
it is far the most important role of strictly economic models 
in normative policy analyses. 

But the same policy maker might also wish to know the 
political implications of pursuing alternative strategies. For 
example, a policy which looks good on economic grounds 
may for various reasons be “politically unacceptable.” 
Game theoretic models can in principle be used to determine 
which policies are politically acceptable in a balance-of- 
power sense. But economic equilibrium models are not game 
theoretic models, and should not be confused as such. 

Finally, there are the distributional concerns which are 
central to normative ethics. Economic planning models in 
and of themselves cannot furnish us with the answer as to 
which policy is most “fair.” President Carter has repeatedly 
emphasized his concern with equity. But in this matter we 
have a problem for ethics—not for price/quantity models. 

While these points may seem too obvious to need explicit 
statement there is often confusion about the proper role of 
economic data in normative policy analysis. For this reason, 
we felt it worthwhile to close with a reminder. There is one 
other important limitation to economic models of the kind 
reviewed above. This concerns the use of an “equilibrium” 
methodology in the context of regulated commodities such 
as natural gas. Many problems of interpretation arise here; 
but they are outside the scope of the present paper. 
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Decision analysis of the 

synthetic fuels commercialization program 


by STEVEN N. TANI 
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Menlo Park, California 

INTRODUCTION 

In his State of the Union Message in January 1975, President 
Gerald Ford called for the accelerated development of U.S. 
energy technology and resources and proposed a compre¬ 
hensive set of energy supply and conservation measures to 
reduce U.S. requirements for imported oil. As one of these 
measures, the President proposed a federal incentive pro¬ 
gram whose goal would be the commercial production of 
one million barrels per day of synthetic fuels by 1985. In 
such a program, the Federal Government would provide 
suitable financial and regulatory incentives to stimulate pri¬ 
vate sector investment in commercial-scale plants to convert 
coal, oil shale, and other relatively abundant domestic re¬ 
sources into clean liquid and gaseous fuels. It was generally 
believed that without such incentives, industry would be 
unlikely to undertake the large risks of synthetic fuel plant 
investments. 

The benefits to be achieved by the synthetic fuels program 
would be the following: 

1. An accelerated accumulation of experience and infor¬ 
mation on the technical, environmental, economic, and 
institutional aspects of commercial-scale synthetic fuel 
production for better-informed private sector invest¬ 
ment decisions. 

2. The development of an industry infrastructure to sup¬ 
port subsequent expansion of the synthetic fuels in¬ 
dustry. 

3. Insurance against high world oil prices and against 
early depletion of domestic sources of conventional 
fuels. 

4. Protection against the losses of an oil embargo. 

5. Improvement in the U.S. international bargaining po¬ 
sition. 

These benefits, however, would be counterbalanced by 
the possible costs of subsidizing synthetic fuels relative to 
less expensive energy sources such as imported oil and other 
domestic resources and by the environmental and socio¬ 
economic costs associated with rapid development of coal 
and oil shale reserves. 

In response to the President’s message, the Interagency 
Task Force on Synthetic Fuel Commercialization was 


formed to evaluate the economic and environmental costs 
and benefits of the program and to recommend to the Pres¬ 
ident a program of appropriate size and scope. The Task 
Force was chaired by the Office of Management and Budget 
and included members from the Federal Energy Adminis¬ 
tration, the Environmental Protection Agency, the Depart¬ 
ments of State, Commerce, and Treasury, the Council on 
Environmental Quality, and the National Science Founda¬ 
tion. We, in the Decision Analysis Group at Stanford Re¬ 
search Institute (now SRI International), were engaged to 
assist the Task Force in conducting an analysis of the pro¬ 
gram. 


STRUCTURE OF THE ANALYSIS 

The fundamental question addressed by this analysis was 
whether the U.S. should have a synthetic fuels commer¬ 
cialization program and, if so, how large the program should 
be. The Task Force defined four distinct program alterna¬ 
tives to be evaluated: 

1. No Program—No federal funding of synthetic fuels 
commercialization but continuation of research and 
development. 

2. Informational Program—-A minimal program designed 
primarily to generate technical, environmental, and 
economic data on various resource-to-fuel conversion 
processes, with synthetic fuel production of about 
350,000 barrels per day by 1985. 

3. Medium Program—A program designed to generate 
more complete information on a wider range of pro¬ 
cesses and to meet the President’s goal of 1,000,000 
barrels per day by 1985. 

4. Maximum Program—A program designed to achieve 
the greatest amount of synthetic fuel production in 1985 
possible without causing major dislocations in the 
economy: 1,700,000 barrels per day. 

The object of the analysis was to determine which of these 
alternatives would be of greatest net benefit to the nation as 
a whole, where net benefit includes both economic and non¬ 
economic impacts. 
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We defined four components of net national benefit for 
the evaluation of the program alternatives: economic impact 
on consumers, economic impact on producers, embargo pro¬ 
tection, and environmental and socio-economic impacts. 

To measure the economic impact on consumers, we uti¬ 
lized the concept of consumer surplus. Consumer surplus is 
the difference between the value of a good to consumers 
and the amount of money they must pay for it. This is shown 
graphically in Figure 1. The demand curve, by definition, is 
the most consumers would pay for each unit of the good, 
which is the value of that unit. If the market price is p, then 
q units will be purchased. For every unit except the last 
one, the value of the good exceeds the price paid for it. The 
shaded area between the price line and the demand curve 
represents the total excess value the consumers receive from 
this good; this is called the consumer surplus. 

In the case of the synthetic fuels program, it was felt that 
a demonstration that synthetic fuels could be produced 
cheaply, if achieved, would have the effect of holding down 
the price of imported oil. The resulting increase in consumer 
surplus would then be credited as a positive benefit of the 
program. 

To measure the impact on producers, we used a concept 
analogous to that of consumer surplus—producer surplus. 
This is the difference between the amount producers receive 
for a good and their marginal cost of producing it. Clearly, 
producer surplus is directly related to the idea of profitabil¬ 
ity. Figure 2 shows producer surplus graphically. The supply 
curve represents the marginal cost of producing each unit of 
the good, which is the least amount of money the producers 
would accept for it. The shaded area between the price line 



QUANTITY 

Figure 1—Consumer surplus 



CAPACITY 

QUANTITY 

Figure 2—Producer surplus 


and the supply curve is equal to the total producer surplus 
for that good. 

We assumed in this analysis that synthetic fuel would be 
a substitute for imported oil. Therefore, if the cost of the 
synthetic fuels turned out to be less than the cost of imported 
oil, the industry would accrue positive producer surplus, 
which would be credited to the program as a benefit. How- 



QUANTITY 
Figure 3—Embargo loss 
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ever, if synthetic fuels turned out to be costlier than im¬ 
ported oil, producer surplus would be negative and industry 
would require a subsidy from the government to cover its 
losses. The amount of this negative producer surplus would 
be charged as a cost of the program. 

The algebraic sum of consumer and producer surplus is 
a measure of the total economic impact of the program on 
the nation assuming normal market conditions. However, it 
does not include the impact of the program in the event of 
an oil embargo. The situation during an oil embargo is illus¬ 
trated in Figure 3. The pre-embargo price and quantity of 
oil are established on the long-term demand curve. If an 
embargo occurs, the quantity of oil available for consump¬ 
tion decreases abruptly. Because of short-term inflexibilities 
in consumption patterns, the marginal value (or shadow 
price) of oil is much higher than the long-term demand curve 
indicates. Here we use a linear short-term demand curve to 
show this effect. The economic cost of the embargo is the 
loss of consumer surplus during the embargo and is repre¬ 
sented by the shaded trapezoidal area. 

The synthetic fuels program, by providing a substitute for 
some of the imported oil, would reduce this embargo loss 
by increasing the amount of fuel available for consumption 


during the embargo. This reduction of embargo loss, 
weighted by the probability of occurrence of an embargo, is 
credited to the program as a benefit. 

Finally, the synthetic fuels program would result in non¬ 
economic costs in the form of environmental damage (e.g., 
increased air pollution) and socio-economic disruption (e.g., 
“boom towns” near mining and conversion facilities). These 
costs, to the extent that they are not internalized in the 
producers’ costs (e.g., pollution control costs), are charged 
to the program. 

The sum of these four components of program impact is 
the measure of net national benefit we used in the analysis 
to evaluate the alternatives. 

Clearly, to determine the net benefit of each program 
alternative, we need to know something about the energy 
supply and demand situation in the future. There was, of 
course, considerable uncertainty about the future energy 
picture, so we used probabilistic modeling techniques to 
quantify and incorporate the uncertainty in the analysis. 

Figure 4 shows the decision tree structure that we ulti¬ 
mately developed for this analysis. We treated the dynamics 
in a simple manner by looking at three discrete time periods. 
In 1975, the government would make its program decision, 



Figure 4 —Decision tree 
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choosing one of the four alternatives. Then, in the mid- 
1980’s, the program would result in information about the 
ultimate cost of synthetic fuels production. Based on this 
information and on the prevailing and projected price of 
imported oil, the industry would make its decision on further 
investment in synthetic fuel plants. The price of imported 
oil would, of course, depend on whether or not the oil 
producers’ cartel remained effective in controlling prices. 
Finally, in the mid-1990’s, when the new synthetic fuel 
plants are on-stream, the program impacts can be deter¬ 
mined by looking at the cost of synthetic fuels, the price of 
imported oil, which again depends on the current state of 
the cartel, and the U.S. energy supply and demand balance. 

The year 1985 was used to typify the decade of the 1980’s 
and the year 1995 to typify the decade of the 1990’s. Program 
cost and benefits were measured in constant 1975 dollars 
and were discounted in 1975 using a discount rate of ten 
percent. 

The decision tree in Figure 4 shows how uncertainty was 
explicitly incorporated in the analysis. Uncertainty about 
each of the factors shown was quantified in the form of 
probabilities. Then, for each combination of factors, which 
defined a unique scenario of the future, both the probability 
of occurrence for that scenario and the discounted net na¬ 
tional benefit associated with it were calculated. Finally, for 
each alternative, the expected net benefit was calculated by 
weighting the outcome of each scenario by its probability 
and summing. Note that the decision tree in Figure 4 defines 
5,832 different scenarios for each of the four program alter¬ 
natives. 

The industry decision in 1985 of how much further in¬ 
vestment to make in synthetic fuels plants required special 
treatment. While the government decision would be made 
on the basis of overall national benefits, the private sector 
decision would be made on the basis of corporate profits 
only. Therefore, in the analysis, the level of corporate in¬ 
vestment that maximized expected future producers surplus 
was selected. 

Figure 5 illustrates the techniques we used to quickly 



1985 IMPORTED OIL PRICE GIVEN A STRONG CARTEL 
(Dollars per barrel) 

Figure 5—Simple encoding technique 
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encode uncertainty in the various factors. The probability 
distribution shown represents uncertainty in the 1985 im¬ 
ported oil price given that the cartel is strong. According to 
this distribution, it is equally likely that the price will be 
above or below $15 per barrel (the median value). Also, 
there is a ten percent chance that the price will be below 
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$11 per barrel (the 10 percent fractile) and a ten percent 
chance that it will be above $19 per barrel (the 90 percent 
fractile). We divided the distribution into three sections hav¬ 
ing areas of 25 percent, 50 percent, and 25 percent and used 
the median value to represent the middle section and the 10 
percent and 90 percent fractiles to represent the two tails. 
Thus, as shown in Figure 6, we said in the analysis that 
there is a 25 percent chance that the 1985 imported oil price 
would be $19 per barrel, a 50 percent chance that it would 
be $15 per barrel, and 25 percent that it would be $11 per 
barrel, given that the cartel is strong. The imported oil price 
given that the cartel is weak is assessed to be much lower— 
$10, $8 and $6 per barrel, respectively. 

Using this simple technique, we encoded the uncertainty 
of the Task Force members about each of the factors shown 
in the decision tree. Of particular interest are the assess¬ 
ments of the future state of the oil producers’ cartel. As 
shown in Figure 7, the chances of the cartel remaining strong 
through 1985 were assessed by the Task Force to be 50-50. 
Given that it is strong in 1985, the probability that it would 
remain strong through 1995 was assessed to be 80 percent, 
while if it is weak in 1985, the chance that it \frould become 
strong by 1995 was assessed to be only 20 percent. 

RESULTS OF THE ANALYSIS 

After we had structured the problem with the decision 
tree, constructed a computer model to calculate the net 
national benefit for each of the thousands of scenarios in the 
tree, and encoded the uncertainties of the Task Force, we 
were ready to compute the analytic results. 

Figure 8 summarizes these results. The total expected 
discounted net benefit (in billions of 1975 dollars) is shown, 
along with its components, for each of the three synthetic 
fuels program levels relative to having no program at all. 
These results indicated that, on balance, the synthetic fuels 
commercialization program was not in the best national in¬ 
terest and that the bigger the program, the greater the na¬ 
tional loss. The small informational program had an ex¬ 
pected impact on minus $1.65 billion. The larger program 
had expected impacts of minus $5.41 billion and minus 
$10.98 billion, respectively. 

We can get more insight by looking at the components of 
total net benefit. While the synthetic fuels program is ex¬ 
pected to have positive impacts on consumer surplus 


Expected Discounted Net Benefit (billions of 1975 dollars) 

Program Alternative 

Consumer 

Surplus 

Producer 

Surplus 

Embargo 

Protection 

Environmental 

and 

Socioeconomic 

Total 

No Program 

0 

0 

0 

0 

0 

Information Program 
(0.35 mm bbl/day) 

1.07 

-2.71 

0.43 

0.44 

-1.65 

Medium Program 
(1 mm bbl/day) 

3.29 

-8.74 

1.18 

-1.14 

-5.41 

Maximum Program 
(1.7 mm bbl/day) 

4.55 

-15.77 

2.23 

-1.99 

-10.98 


Figure 8—Expected program impacts 
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Figure 9—Uncertainty in program impacts 

through the possible moderation of future imported oil prices 
and on embargo losses through a slight reduction in oil 
imports, these benefits are far outweighed by the negative 
impact on producer surplus. Basically, it was far more likely 
than not that synthetic fuels would be more expensive than 
imported oil and therefore need a subsidy. The negative 
impact of environmental and socio-economic costs is rela¬ 
tively minor. 

The results shown in Figure 8 are the expected values of 
program impacts. There is, of course, considerable uncer¬ 
tainty about the impact of the program, as shown in Figure 
9. While the expected impact of the informational program 
is $1.65 billion, there is a 30 percent chance that the net 
impact will be positive and a 10 percent chance that it will 
be as much as +$7 billion. On the other hand, there is a 10 
percent chance that it will be as negative as -$9 billion. It 
is equally likely that the impact will be worse than or better 
than — $4 billion. The uncertainty in the impact of the larger 
program is even greater. 

Figure 10 is a partial expansion of the decision tree that 
shows how two of the factors affect the results of the anal¬ 
ysis. The -$1.65 billion expected impact of the information 
program consists of a 50 percent chance of —$4.86 billion if 
the cartel in 1985 is weak and a 50 percent chance of +$1.55 
billion if it is strong. Note that a weak cartel, which leads 
to generally lower imported oil prices, is bad for the syn- 
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thetic fuels program while presumably very good for the 
nation as a whole. Conversely, a strong cartel, with higher 
imported oil prices, makes the program look good while 
being bad for the nation. This emphasizes that the synthetic 
fuels program is a hedging strategy—it pays off when other 
things are going badly. Note also that if the cartel is weak, 
the program looks bad even if synthetic fuels turn out to be 
cheap to produce. On the other hand, if synthetic fuels turn 
out to be expensive, the program looks bad even if the cartel 
is strong. That is why, on balance, the program looks bad. 

The assessment of a 50-50 chance that the cartel would 
remain strong through 1985 turned out to be pivotal and 
more than a little controversial within the Task Force. To 
show the implications of different probabilities for this fac¬ 
tor, we performed a sensitivity analysis, which is shown in 
Figure 11. This gives the expected net impact of each pro¬ 
gram level relative to no program as a function of the prob¬ 
ability of a strong cartel in 1985. It assumes that with 80 
percent probability, the cartel will remain in the same state 
from 1985 to 1995. Figure 11 shows that only if the proba¬ 
bility of a strong cartel in 1985 exceeds 75 percent does the 
information program look better than no program and that 
the probability must exceed 82 percent for the medium size 
program to be the best alternative. An interesting result is 
that the maximum size program is never optimal for any 
value of this probability. 

So far, the analytic results have been presented only in 
terms of expected values. It might be argued that the deci¬ 
sion should not be made on the basis of expected values but 
rather on the basis of values that are adjusted for risk. To 
show how various levels of risk aversion would affect the 



Figure 11—Sensitivity to cartel probabilities 



results, we prepared the sensitivity profile shown in Figure 
12. Here, we have assumed that the nation’s risk attitude is 
expressed by one of the family of exponential utility curves. 
The degree of risk aversion expressed by this curve is given 
by its one parameter, called the risk tolerance; the smaller 
the risk tolerance, the greater the degree of risk aversion. 
In personal terms, an individual’s risk tolerance is the largest 
amount of money he would willingly risk in a gamble that 
is equally likely to halve or double that amount. 

Figure 12 shows the value to the nation of each program 
level relative to no program as a function of the nation’s risk 
tolerance assuming an industry risk tolerance of $5 billion 
for the private sector capacity expansion decision. Note first 
that the value of the program increases as the nation’s risk 
aversion increases. This is characteristic of a hedging strat¬ 
egy, since it reduces overall uncertainty. However, the na¬ 
tion’s risk tolerance must be less than $67 billion for the 
information program to be better than no program and it 
must be less than $56 billion for the medium size program 
to be the best alternative. We believe that a reasonable range 
for the nation’s risk tolerance is from one-fourth to one-half 
of annual GNP, or about $300 billion to $600 billion. As 
Figure 12 shows, for any risk tolerance in this range, the 
ranking of program alternatives is the same as in the ex¬ 
pected value case, with the best alternative being no pro¬ 
gram at all. 

EPILOGUE 

As far as we can determine, this is the first decision 
analysis to be presented in the White House. The chairman 
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of the Task Force presented it to the President’s Energy 
Resources Council in July 1975. Citing benefits of the pro¬ 
gram that were not quantified in the analysis, such as the 
international leverage gained by the U.S. in asserting posi¬ 
tive leadership in developing alternate fuel sources, as well 
as the “relatively small risk and expected cost” of the small 
program, the Task Force recommended that the government 
undertake the informational program alternative with a pos¬ 
sibility that it could switch to the medium size program 
pending additional information on crucial factors. The Ad¬ 
ministration’s bill incorporating this recommendation ulti¬ 
mately failed to pass through Congress. 

COMPUTER UTILIZATION 

For this analysis, we wrote a FORTRAN program for use 
on a commercial time sharing system. The program, which 
we created from scratch rather than use off-the-shelf rou¬ 


tines, required approximately 400 lines of code and cost 
roughly fifteen dollars to run a complete evaluation of the 
decision tree. 

One feature of the time sharing service that we found to 
be especially useful was the accessibility from different lo¬ 
cations. We did most of the model development at SRI 
headquarters in Menlo Park, California, but we used the 
program extensively while working with the Task Force in 
Washington D.C. Indeed, one of the most valuable aspects 
of our assistance to the Task Force was our ability to answer 
almost immediately their many questions about the effect of 
changes on the assumptions and assessments in the analysis. 
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INTRODUCTION 

One of the critical factors affecting the electric utility and 
electrical manufacturing industries is the forecast of the 
long-term demand for electricity. Superficial forecasts based 
on either past time trends or very broad based factors for 
the entire United States do not provide sufficient confidence 
for long-range planning. This effort is an attempt to examine 
the basic factors in the U. S. which create the demand for 
electricity. Equally, or possibly even more important, is the 
examination of the sensitivity of the forecast to the basic 
driving factors. Through this sensitivity analysis, which re¬ 
lates the changes in input assumptions to output changes, 
the reasonableness of other forecasts can be established. 

POWER SYSTEMS MARKET FORECAST MODEL 

When one begins the development of a market forecast 
model, some important basic concepts must be resolved. 1 
The time span of interest in the forecast must be specified. 
Whether the span is hours, days, months, years, or decades 
will influence the structure selected for the model. In gen¬ 
eral, for short time spans, the model structure can be closely 
patterned after the existing market. For long-term models, 
however, the dynamics of a changing market must be rep¬ 
resented in the model. 

Because of different climates, industries, economics, and 
traditions, geography plays an important role in an electric 
demand forecast model. The differing regional saturations 
of electric air conditioning and space heating, mixes of in¬ 
dustry, and price of electricity compared with competing 
fuels require that geographical considerations be included to 
obtain better forecasts. 

Nine data bases are presently maintained for each of the 
nine NERC regions. These regions are geographically lo¬ 
cated as shown in Figure 1. One obviously questions the 
selection of these regions versus some other delineation. 
When dealing with demand data, such as population, house¬ 
holds, value added, etc., a state or census region division 
might seem more reasonable. However, when dealing with 
electric supply data, such as generation capacity, additions, 
costs, etc., an electric utility boundary or power pool area 


seems reasonable. The basic problem is that data collection 
for supply and demand is on a completely different geo¬ 
graphic basis. Each NERC region is a grouping of Power 
Supply Areas which are, in turn, a grouping of utilities, 
many of which overlap state and political boundaries. To 
establish consistency in the data base, we decided that de¬ 
mand data given at the state level could be more easily 
transformed to a supply area, such as an NERC region, 
rather than vice versa. The NERC supply region was se¬ 
lected rather than the power supply areas or utility service 
areas simply because the model detail we were requiring 
would need excessive data storage and manipulation at the 
more disaggregated level. 

A block diagram of the model is shown in Figure 2. Sub¬ 
models for residential, industrial, and commercial are used 
to estimate kilowatthour (KWH) sales. Exogenous estimates 
of transportation, agricultural, governmental, and miscella¬ 
neous KWH sales are summed with residential, commercial, 
and industrial to obtain total sales. Total electrical energy 
output is calculated using transmission and distribution loss 
estimates determined from input data specifying losses as a 
percent of KWH sales. The peak load in each region is 
calculated from total output using a load factor which is 
estimated as a function of the mix of KWH sales. The 
installed capacity in each region is calculated from peak load 
using either an option that inputs the percentage of genera¬ 
tion additions of each type or an option that uses generation 
capital, fuel price, and investment data to determine the 
least cost mix of additions. The capacity factor for each unit 
type is calculated by adjusting the KWH output from all 
units (capacity factor x installed capacity rating x 8760 
hrs.) to be equal to the estimated KWH generation from the 
model. Input energy to all types of units is calculated using 
the estimated electrical generation and the heat rate for each 
type of unit. Additional detail on each of the calculational 
blocks and models depicted in Figure 2 is shown in Figures 
3 through 6 and is discussed in the following. 

Residential sector model 

A block diagram of the model used to estimate residential 
KWH sales for each NERC region is shown in Figure 3. By 
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multiplying population by headship rate, an estimate of 
households is obtained. Time trends for the saturation of 
appliances, air conditioning, and space heating are input to 
the model 2 and are used to calculate saturations in any given 
year. These saturations are then adjusted based on price 
effects and multiplied by the number of households to obtain 
an estimate of the number of electrical end-use units. Av¬ 
erage annual use for each unit type is estimated by inputting 
time trends of annual use and adjusting annual use based on 
price effects. An estimate of annual KWH consumption is 
obtained by multiplying the number of end-use units by the 
annual use. 

The three equations used to estimate residential KWH 
sales in any given year are: 

RKWH = 2 POP*HDR*SAT (1) 

all appl- 

*SATPA*USE*USEA 


( \ aSe ^! / \ 

2 RPE/LOA) ( 2 RPG/LOA) (2) 

LOA / \LOA / 

USEA=RPE aUE (3) 

where POP is regional population, HDR is regional headship 
rate, SAT is appliance saturation, SATPA is price adjust¬ 
ment of saturation, USE is annual KWH consumption, 
USEA is the price adjustments of annual use, RPE and RPG 
are the relative prices of electricity and gas with respect to 
the base year, LOA is the lifetime of an average appliance, 
«se and « SG are electric and gas price elasticities used to 
adjust saturation, and a UE is the electric price elasticity used 
to adjust annual use. 

Industrial sector model 

The industrial sector model, shown in Figure 4, uses Fed¬ 
eral Reserve Board production indices as a national produc¬ 
tion indicator for two digit manufacturing and mining cate¬ 
gories. Projected trends in dollars of value added by SIC 
code are used to calculate ratios which estimate the share 
of national production to be allocated to each NERC region 
in any given year. Input time trends are used to estimate 
electricity use per unit of production for each SIC code. 
Industrial KWH consumption is estimated using the regional 
production estimates and the energy intensiveness estimates 
corrected for price effects. 

The equations used to estimate industrial KWH consump¬ 
tion are given by: 

INDKWH= 2 FRB nat *($VAREGSIC/$VANATSIC) 

SIC 

*USESIC*IUSEA t (4) 

IUSEA t =(l -XjlUSEAt-i+RPEI^RPOI^ 10 (5) 


Demographic Data 

Fuel & Electricity 
Prices 

Elasticities 
Appliance Lifetimes 


Energy Prices 
Intensity Trends 
Elasticities 
Production Indices 
Regional Coefficients 
Equipment Lifetimes 


Energy Prices 
Regional Coefficients 
NRS Growth Rate 
Equipment Lifetimes 
SP. HTG., AC, & Light 
Saturations & 
Intensities 



Figure 2—Power systems market forecast model 
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Figure 3—Residential sector model 


where FRB NAT is the Federal Reserve Board Indices, 
$VAREGSIC is the $ of value added by SIC by region, 
$VANATSIC is the $ of value added nationally by SIC, 
USESIC is the electricity use per unit output for each SIC, 
IUSEA t and IUSEA t _ 1 are the industrial use price adjust¬ 
ments in years t and t—1, X is the fraction of this price 
adjustment accomplished each year, RPEI and RPOI are the 
relative prices of electric and other fuels in the industrial 
sector to a base year, « UIE and aui 0 are the price elasticities 
of use for electric and a composite for other fuels. 

Commercial sector model 

The commercial sector model, shown in Figure 5, uses 
national projections of non-residential structures or com¬ 
mercial buildings to estimate commercial floor space in use 


in any given year. Regional allocators developed from re¬ 
gional population and personal income forecasts 3 are used 
to estimate floor space in each NERC region. Input time 
trends are used to calculate both the lighting, space heating, 
air conditioning, and miscellaneous electricity use per 
square foot of floor space and the saturation of these uses. 
A price adjustment is made on total commercial demand. 
The equations used for the estimated KWH consumption 
are: 

COMKWH=NRSGR*COMFLB* REGAL 

* 2 USEC*CSAT*COMPA t (6) 

COMPA t =(l -X c )COMPA t _i 

+X c RPEC QCE *RPGC OCG (7) 

where NRSGR is national non-residential structures or com- 
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Figure 4 —Industrial (mining & manufacturing) sector model 
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Figure 5—Commercial sector model 


mercial buildings growth index relative to the base year. 
COMFLB is the commercial square feet of floor space in 
the base year, REGAL is the fraction of floor space allocated 
to each region, USEC is the KWH consumption per square 
foot of floor space for each specific use, COMPA t and 
COMPAt-j are the commercial price adjustments in year t 
and t— 1, \ c is the fraction of the price adjustment taking 
place each year, RPEC and RPGC are the relative prices of 
electricity and gas with respect to the base year, and a CE 
and a CG are the price elasticities of total use in the commer¬ 
cial sector. 

Load factor model 

Ideally, one would like to estimate and forecast the peak 
electric load independently from KWH sales. However, be¬ 
cause of the scarcity of data with regard to individual loads 
and their coincidence on the system, this is impossible to 
accomplish in general. The approach used here is to relate 
peak load to residential, commercial, and industrial KWH 
sales, saturation of space heating and air conditioning, cool¬ 
ing and heating degree days. After obtaining peak load 
(PLW, PLS), the load factors (LFS, LFW) are calculated 
by relating to the total KWH output. The equations used are 

PLW=a 0) RKWH+/3 w COMKWH+yJNDKWH 

+ o- tt RHS + 5 w HDD (8) 

PLS=a s RKWH+&COMKWH+y s INDKWH 

+<x s RAC+ 5 s CDD (9) 


LFS=(KWHOUT-(8760XPLS)) (10) 

LFW=(KWHOUT)h- ( 8760 X PLW)) (11) 

where PLW and PLS are winter and summer peak load, 
LFS and LFW are summer and winter load factor, 
KWHOUT is total KWH output, RSH is the saturation of 
residential electric space heating, RAC is the saturation of 
residential air conditioning, HDD and CDD are the regional 
average heating and cooling degree days. 

The coefficients in equations (8) through (10) are estimated 
from historical data for each region. These independent var¬ 
iables were chosen because in the past, residential and com¬ 
mercial loads have contributed more to peak than industrial 
loads. Also, loads with larger weather-sensitive components 
have exhibited generally lower load factors. 

Generation capacity model 

Using total KWH output and regional load factor, peak 
load is computed for each region for each year of interest. 
With an estimate of peak load as input, the function of the 
generation capacity model is to determine how much capac¬ 
ity (if any) should be added in each region during each time 
period. If additions are required, then the model must cal¬ 
culate the type needed from a group of nine possible unit 
types: fossil (coal, oil, gas), nuclear, combustion turbine, 
combined cycle, hydro, pumped storage, and other. 

The block diagram shown in Figure 6 demonstrates the 
functions performed by the generation capacity model. An 
initial estimate of the end-of-year capacity is calculated by 
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BOY = Beginning of Year 
EOY = End of Year 

Figure 6—Generation capacity model 


adding backlog, delays, and adjustments and subtracting 
retirements from the beginning-of-year capacity. Using this 
estimate of end-of-year (EOY) capacity with inputs of peak 
load and the minimum reserve margin in the region, a check 
is made to establish the sufficiency of the capacity. If the 
capacity is insufficient to meet peak load requirements with 
the minimum reserve margin, then new orders are generated 
for additional units. These new orders are then added to the 
initial estimate of the installed capacity to obtain the in¬ 
stalled EOY capacity. These new order units are then 
checked against the equipment lead times to establish that 
sufficient time is available so that units being added can be 
delivered and installed. Gross additions for each type unit 
are calculated by adding backlog, delays, and new orders. 

There are two options in the new orders model for cal¬ 
culating the type of units to be added. The first option simply 
provides for calculating the mix of new units using percent¬ 
ages estimated from the NERC reports. The second selects 
additional capacity from the nine types of units so that 
additions are matched to load requirements in a manner that 
minimizes cost. 


BASE ELECTRICAL ENERGY AND PEAK LOAD 

FORECAST 

To demonstrate the capabilities of the model, a base 
forecast has been prepared which uses the nine NERC data 
bases and aggregates the results from these regions to obtain 
a national forecast (Tables I, II and III). Because of space 
limitations, very little of the model output can be shown. 
Only a very brief summary of the base case is presented 
here. 


The macroeconomic factors which drive the model are 
taken from the output of a macroeconomic model 4 so that 
consistency is established. For example, the industrial pro¬ 
duction, household growth, and commercial buildings 
growth are all consistent with a much larger set of economic 
conditions. The number of variables used directly as input 
to the electrical demand model from the macroeconomic 
model output is small (~25 time series). However, the ma¬ 
croeconomic model which derives these time series is com¬ 
prehensive (over 1100 equations and identities), and, there¬ 
fore, indirectly the influence of all pertinent economic 
factors are included. 

National and regional KWH sales and output (sales+ 
losses) are shown in Table I; historically for 1971-1976 and 
forecasted 1977-2000. Compounded annual growth in the 
period 1977-2000 is approximately 5 percent. This is consid¬ 
erably less than historic growth in the decades of the 60s 
and 70s where annual growth averaged 7 percent. The in¬ 
dustrial sector is the major contributor to growth in this 
forecast based on the compounding effects of long-term ex¬ 
pansion and increasing use of electricity per unit output in 
most industry groups. All NERC regions exhibit lower 
growth rates than historic and range from approximately 5.5 
percent in the southwest to 3 percent in the northeast. 

National and regional peak loads as forecasted by the 
model are shown in Table II. Actual values for the period 
1971-1976 are shown in parentheses. Nationally, summer 
peak loads are forecasted to grow at about 4.6 percent an¬ 
nually and winter peak about 4.8 percent. Thus, peak is 
forecasted by this model to grow at a lower rate than KWH 
output. This can be explained by the major part of KWH 
output growth being industrial which contributes less to 
peak, and the application of load management techniques in 
controlling peak. Regionally, peak growth is highest in the 
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TABLE I.—National and Regional KWH Sales and Output 

KILOWATT HOUR OUTPUT-BILLIONS 


YEAR 

NPCC 

MAAC 

ECAR 

SERC 

MAIN 

SPP 

ERCOT 

MARCA 

wscc 

TOTAL 

1971 

167.600 

138.200 

278.000 

318 . 300 

129.800 

121.700 

94.100 

61.500 

307.900 

1,617.100 

1972 

178.100 

147.400 

303.200 

346.700 

138.400 

132.800 

104.700 

66.400 

332.200 

1,749.900 

1973 

189.000 

158.500 

325.400 

384.400 

146.400 

140.700 

110.400 

69.900 

344 . 300 

1,869.000 

1974 

184.700 

154.300 

318.400 

384.400 

151.000 

144.400 

113.800 

73.800 

350.700 

1,875 . 500 

1975 

183.800 

154.400 

319.500 

396.300 

155.200 

156.600 

115 . 100 

77.100 

362.200 

1,920.200 

1976 

191.800 

161.100 

341.800 

426.800 

160.400 

168.700 

120.800 

77.400 

389.500 

2,038.300 

1977 

202.800 

171.700 

368.300 

458.400 

173.200 

180.800 

129.900 

82.500 

420.000 

2,187.600 

1978 

213.000 

181.600 

390.800 

490.500 

185.100 

192.700 

1 38.900 

87.600 

448 . 500 

2,328.700 

1979 

222.600 

191.100 

412.100 

521.500 

196.500 

204.500 

148.000 

92.400 

476.000 

2,464.700 

1980 

232.200 

200.700 

435.700 

551.600 

208.700 

216.600 

157.600 

97.400 

505 . 500 

2,606.000 

1985 

278.900 

250.500 

547.000 

721.200 

269.200 

282.700 

211.100 

122.800 

661.200 

3,344.600 

1990 

325.200 

307.900 

667.800 

943.200 

342.900 

366.900 

293.400 

152.000 

854.000 

4,253.300 

1995 

372.000 

371.800 

801.600 

1,221.900 

428.200 

470.400 

384.900 

184.900 1, 

,084.000 

5,319.700 

2000 

413.400 

435.000 

942.300 

1,550.000 

522.700 

595.000 

490.400 

220.100 1. 

, 349 .000 

6,517.900 



KILOWATT 

HOUR SALES- 

BILLIONS 








YEAR 

NPCC 

MAAC 

ECAR 

SERC 

MAIN 

SPP 

ERCOT 

MARCA 

WSCC 

TOTAL 

1971 

154.700 

124.000 

250.000 

288.000 

117.100 

109.900 

84.700 

55.200 

283.000 

1,466.600 

1972 

161.700 

132.100 

272.000 

313.700 

124.200 

119.800 

92.700 

59.200 

302.400 

1,577.800 

1973 

182.200 

142.000 

292.900 

350.400 

131.600 

127.100 

99.400 

63.500 

314.300 

1,703.400 

1974 

167.300 

139.600 

288.600 

351.000 

136.000 

129.400 

101.900 

66.600 

320.300 

1,700.700 

1975 

165.600 

137.500 

287.400 

360.900 

138.700 

140.900 

104.000 

69.200 

328.600 

1,732.800 

1976 

172.300 

146.500 

311.000 

388.400 

147.300 

151.700 

110.500 

69.400 

345.900 

1,843.000 

1977 

182.200 

156.100 

335.100 

417.100 

159.000 

162.700 

118.800 

73.900 

373.000 

1,977.900 

1978 

191.400 

165.100 

355.600 

446.300 

170.000 

173.600 

127.100 

78.500 

398.300 

2,105.900 

1979 

200.000 

173.700 

375.000 

474.500 

180.400 

184.400 

135.400 

82.800 

422.800 

2,229.000 

1980 

208.600 

182.400 

396.500 

501.900 

191.700 

195.500 

144.200 

87.300 

448.900 

2,357.000 

1985 

250.600 

227.700 

497.800 

656.300 

247.200 

256.300 

193.200 

110.000 

587.200 

3,026.300 

1990 

293.000 

280.400 

607.600 

858.200 

314.900 

332.100 

255.600 

136.200 

758.500 

3,836.500 

1995 

335.200 

338.600 

729.400 

1,111.800 

393.200 

425.700 

335.600 

165.700 

962.800 

4,798.000 

2000 

372.500 

396.100 

857.400 

1,410.400 

480.000 

534.400 

428.900 

197.300 1, 

,198.100 

5,875.100 
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YEAR 

NPCC 

MAAC 

ECAR 

SERC 

MAIN 

SPP 

ERCOT 

MARCA 

WSCC 

TOTAL 

1971 

29.315 

25.991 

45.571 

57.973 

26.410 

24.654 

18.839 

12.034 

53.365 


294.152 

1972 

29.786 

27.217 

48.987 

62.541 

27.327 

26.920 

20.971 

12.918 

56.564 


313. 231 

1973 

34.595 

30.152 

53.409 

68.915 

29.417 

28.700 

22.857 

13.891 

58.755 


340.693 

1974 

32.044 

29.808 

52.962 

69.352 

29.953 

29.379 

23.553 

14.598 

59.925 


341.573 

1975 

32.255 

30.128 

53.666 

72.196 

30.643 

31.990 

24.223 

15.158 

62.178 


352.438 

1976 

33.479 

31.537 

56.756 

76.114 

3'.488 

34.611 

2 5.99 8 

15.125 

65.979 


371.087 

1977 

35.483 

33.813 

61.268 

81.672 

34.173 

37.440 

28.204 

16.305 

70.090 


3 9R.450 

1978 

37.283 

35.565 

64.662 

86.499 

36.072 

40.261 

30.426 

17.476 

74.303 


422.547 

1979 

38.996 

37.256 

67. Q 78 

91.150 

37.955 

43.040 

32.633 

18.619 

78.381 


446.007 

1980 

40.724 

38.980 

71.543 

95.609 

39.870 

45.993 

34.°66 

10.761 

8 2.5 8 8 


469.986 

1985 

49.357 

47.976 

88.480 

120.257 

49.621 

61.830 

48.098 

25.731 

105.031 


596.381 

1990 

57.911 

57.986 

105.051 

150.644 

60.816 

81.816 

64.993 

32.618 

131.659 


744.397 

1995 

66.447 

68.611 

124.934 

187.094 

73.188 

106.544 

86.523 

40.384 

162.790 


916.517 

2000 

74.452 

79.126 

144.712 

228.774 

86.632 

135.378 

111.857 

48.719 

197.R99 

1, 

,107.549 


WINTER-PEAK 

LOAD (GW) 










YEAR 

NPCC 

MAAC 

ECAR 

SERC 

MAIN 

SPP 

ERCOT 

MARCA 

WSCC 


TOTAL 

1971 

29.685 

20.468 

43.514 

51.487 

20.235 

16.328 

11.736 

10.378 

55.051 


258.881 

1972 

30.969 

22.501 

48.356 

57.386 

22.290 

18 . 533 

13.958 

12.036 

58.240 


284.270 

1973 

34.174 

23.738 

51.395 

64.286 

22.684 

20.740 

15.710 

12.088 

60.792 


305.606 

1974 

32.488 

24.308 

51.623 

65.613 

24.384 

21.502 

16.737 

13.234 

61.453 


311.342 

1975 

32.720 

24.647 

52.275 

69.104 

25.081 

24.488 

17.609 

14.140 

63.979 


324.043 

1976 

34.227 

26.918 

56.764 

74.563 

26.826 

26.685 

19.064 

14.001 

67.444 


346.491 

1977 

36.072 

28.726 

60.991 

80.455 

29.071 

30.079 

21.436 

15.309 

72.654 


374.792 

1978 

38.138 

30.798 

64.884 

86.323 

31.358 

33.252 

23.605 

1 6.477 

77.450 


402.286 

1979 

40.123 

32.777 

68.682 

92.003 

33.591 

36.428 

25.754 

17.669 

82.117 


429.145 

198 0 

42.112 

34.783 

72.749 

97.545 

35.914 

39.627 

27.990 

18.825 

86.922 


456.467 

1985 

51.859 

44.839 

92.151 

127.068 

47.091 

54.001 

39.466 

24.573 

112.623 


593.670 

1990 

61.428 

56.079 

112.259 

163.235 

59.874 

70.397 

53.446 

30.721 

142.975 


750.416 

1995 

70.962 

68.017 

133.994 

206.004 

73.999 

88.677 

65.181 

37.289 

178.112 


925.235 

2000 

79.949 

79.725 

156.692 

254.639 

89.292 

106.150 

85.577 

44.066 

217.732 

1 , 

,113.821 
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TABLE III.—National and Regional Load Factors 

LOAD FACTOR - SUMMER PEAK 


YEAR 

NPCC 

MAAC 

ECAR 

SERC 

MAIN 

SPP 

ERCOT 

MARCA 

WSCC 

NAT 

1971 

0.653 

0.607 

0.696 

0.627 

0 . 561 

0.564 

0.570 

0. 583 

0.659 

0.628 

1972 

0.683 

0.618 

0.707 

0.633 

0.578 

0.563 

0.570 

0.587 

0.670 

0.638 

1973 

0.624 

0.600 

0.696 

0.637 

0.568 

0.560 

0.551 

0. 574 

0.669 

0.626 

1974 

0.658 

0 . 591 

0.686 

0.633 

0.575 

0.561 

0.552 

0.577 

0.668 

0.627 

1975 

0.650 

0.585 

0.680 

0.677 

0.578 

0.559 

0 . 542 

0.5P1 

0.665 

0.622 

1976 

0.654 

0.583 

0.687 

0.64C 

0.582 

0.556 

0.530 

0.584 

0.674 

0.627 

1977 

0.652 

0.580 

0.686 

0.641 

0.579 

0.551 

0.526 

0.578 

0.684 

0.627 

1978 

0.652 

0.583 

0.690 

0.647 

0.586 

0.546 

0.521 

0.572 

0.689 

0.629 

1979 

0.652 

0.586 

0.692 

0.653 

0.591 

0.542 

0.518 

0.567 

0.693 

0.631 

1980 

0.651 

0.588 

0.695 

0.659 

0. 598 

0.538 

0.515 

0.563 

0.699 

0.633 

1985 

0.645 

0.596 

0.706 

0.685 

0.619 

0.522 

0.501 

0.545 

0.719 

0.640 

1990 

0.641 

0.6C6 

0.720 

0.715 

0.644 

0.512 

0.515 

0.532 

0.740 

0.652 

1995 

0.639 

0.619 

0.732 

0.746 

0.668 

0.504 

0.508 

0.523 

0.760 

0.663 

2000 

0.634 

0.628 

0.743 

0.773 

0.689 

0.502 

0.500 

0.516 

0.778 

0.672 


LOAD FACTOR-WINTER PEAK 


YEAR 

NPCC 

MAAC 

ECAR 

SERC 

MAIN 

SPP 

ERCOT 

MARCA 

WSCC 

NAT 

1971 

0.645 

0.771 

0.729 

0.706 

0.732 

0.851 

0.915 

0.677 

0.638 

0.713 

1972 

0.657 

0.748 

0.716 

0.690 

0.709 

0.818 

0.856 

0.630 

0.651 

0.703 

1973 

0.631 

0.762 

0.723 

0.683 

0.737 

0.774 

0.802 

0.660 

0.647 

0.698 

1974 

0.649 

0.725 

0.704 

0.669 

0.707 

0.767 

0.776 

0.637 

0.651 

0.688 

1975 

0.641 

0.715 

0.698 

0.655 

0.706 

0.730 

0.746 

0.622 

0.646 

0.676 

1976 

0.640 

0.683 

0.687 

0.653 

0.683 

0.722 

0.723 

0.631 

0.659 

0.672 

1977 

0.642 

0.682 

0.689 

0.650 

0.680 

0.686 

0.692 

0.615 

0.660 

0.666 

1978 

0.638 

0.673 

0.688 

0.649 

0.674 

0.662 

0.672 

0.607 

0.661 

0.661 

1979 

0.633 

0.666 

0.685 

0.64 7 

0.668 

0.641 

0.656 

0.597 

0.662 

0.656 

1980 

0.629 

0.659 

0.684 

0.646 

0.663 

0.624 

0.643 

0.591 

0.664 

0.652 

1985 

0.614 

0.638 

0.678 

0.648 

0.653 

0.598 

0.611 

0.570 

0.670 

0.643 

1990 

0.604 

0.627 

0.679 

0.660 

0.654 

0.595 

0.627 

0.565 

0.682 

0.647 

1995 

0.598 

0.624 

0.683 

0.677 

0.661 

0.606 

^ 0.644 

0.566 

0.695 

0.656 

2000 

0.590 

0.623 

0.686 

0.695 

0.668 

0.640 

‘0.654 

0.570 

0.707 

0.668 


Southwest (6.2 percent) and lowest in the mid-Atlantic (3.5 
percent) and Northeast (3.5 percent). 

National and regional load factors are shown in Table III. 
Nationally, summer load factor is forecasted to improve for 
the same reasons mentioned previously, that summer peak 
load is growing less than KWH output, industrial growth, and 
load management. National winter load factor is forecasted 
to decline because of the considerable increase in electric 
space heating, especially in the more moderate climates. 

The low summer load factors forecasted by the model for 
ERCOT, SPP, and MARCA may not actually occur because, 
most probably, additional load management equipment and 


incentives will be applied to arrest these declining load fac¬ 
tors. 
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Computer modeling of automotive engine combustion 
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INTRODUCTION 

Using a general outline of the physical and chemical events 
which occur in an automobile combustion chamber during 
a complete engine cycle, we discuss some of the achieve¬ 
ments and some of the limitations of computer modeling of 
automotive engine combustion. We point out that this type 
of modeling is often limited by lack of physical knowledge 
and by computer software and hardware restrictions. Still, 
within these constraints many types of combustion models 
have been able to make important contributions to present 
understanding of automobile engine combustion. 

Although the internal combustion engine has been in use 
in automobiles for about a century, the current need for 
increased fuel economy and reduced chemical pollutant 
emissions creates many new problems for automotive engine 
design. In order to provide support for engine designers and 
to provide additional theoretical analysis and insight into the 
physical and chemical processes which are part of engine 
combustion, increased attention has recently been given to 
detailed computer modeling of automotive engine combus¬ 
tion. 

Computer models of combustion processes are limited in 
three principal areas. The first of these areas involves in¬ 
adequacies in fundamental descriptions of relevant physical 
systems. For example, fluid mechanical turbulence has a 
profound influence on many aspects of internal combustion 
engine performance, but the theory of turbulence and its 
effects on chemical reactions and flame propagation are 
rudimentary at present. In addition, theoretical descriptions 
of processes such as liquid atomization and other multiphase 
flow systems are generally inadequate. The second major 
limitation for numerical models of complex physical and 
chemical systems is that, even when the physical description 
is relatively complete, often there are serious deficiencies in 
the numerical methods available for solution of the relevant 
systems of equations. The comparatively recent develop¬ 
ment of “stiff equation” solution methods has made a sig¬ 
nificant improvement in the ability to handle the chemical 
kinetic rate equations of combustion chemistry, but other 
problems remain. In computations which include the effects 
of spatial transport of mass and energy, there are very large 
systems of coupled partial differential equations to be 
solved, often between 25 and 50 equations for each of as 


many as 10000 computational cells, for a total of nearly half 
a million differential equations. Matrix solution techniques 
for such systems are not available and most problems must 
be greatly simplified in order to find solutions. The final 
major limitation for many numerical models is the result of 
basic computer hardware constraints, including available 
core size, memory access times, and arithmetic operation 
speeds. These limitations restrict the spatial resolution, 
chemical kinetic detail, and geometrical generality of the 
problems which can be addressed with computer models. 
The combustion models which have been developed to date 
are all subject to each of these major limitations, and it is 
unlikely that this situation will change substantially in the 
near future. Still, within these limits, a great variety of useful 
and valuable information has been developed through nu¬ 
merical modeling of combustion problems. 

In the internal combustion engine, fuel is introduced into 
the combustion chamber in any of a variety of forms, in the 
gas phase, as a liquid, or a combination of the two phases. 
The resulting mixture of fuel and air is then ignited, again 
in any of several methods, and the resulting flame or flames 
propagate through the combustion chamber, consuming fuel 
and oxygen and producing, as products of combustion, 
water, carbon dioxide, and an often bewildering variety of 
chemical pollutants. The energy released by the reactions is 
converted to mechanical energy as the piston moves down¬ 
ward. The flame can be extinguished by a number of mech¬ 
anisms, by contact with the cold walls of the combustion 
chamber, by the rapid expansion of the combustion chamber 
as the piston moves downward, or by consumption of avail¬ 
able fuel. Many forms of flame extinction or quenching leave 
unburned fuel in the combustion chamber which can even¬ 
tually result in hydrocarbon pollutants in the atmosphere. 
The combustion products are then passed through an ex¬ 
haust system where the product gases are often subjected 
to additional reaction environments, such as catalytic con¬ 
verters, in order to modify the exhaust composition. Seen 
in this general outline, it is readily apparent that no single 
numerical model can be expected to deal adequately with 
the physical and chemical details of the entire process. One 
approach to modeling the combustion process is that which 
is characterized as thermodynamic models. 1 In these models 
the gases are divided into two regions, burnt and unburnt; 
empirical relations are then used to determine the rate at 
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which gases go from unburnt to burnt. These models provide 
little information on the spatial and chemical details of the 
combustion process. Another approach is to break the broad 
total problem into a sequence of overlapping smaller prob¬ 
lems which can be analyzed separately. Only by this type 
of fragmentation can any of the physical subsystems be 
made tractable for analysis. It is also common to use sym¬ 
metry assumptions to simplify the geometrical aspects of 
many problems, since fully three-dimensional problems usu¬ 
ally require a prohibitive amount of computer memory and 
CPU time. 

The problem of fuel injection into a combustion chamber 
provides an instructive example of a number of these points. 
Numerical models of liquid fuel sprays 2-4 have assumed that 


the liquid spray consists of an array of independent spherical 
droplets. This type of treatment ignores the important proc¬ 
ess of the atomization of the initially continuous liquid jet, 
since there is no physical theory which properly describes 
liquid jet atomization. When the assumption is made that 
the jet does break up into individual fuel droplets, a purely 
computational problem arises. In one of the more ambitious 
droplet models to date, 2 the liquid droplet distribution func¬ 
tion becomes a variable in eight independent dimensions, 
including three space coordinates, three velocity coordi¬ 
nates, the droplet radius, and the time. With even the min¬ 
imum resolution in each dimension, this treatment requires 
the computational manipulation of a variable with a dimen¬ 
sion of more than 1.5 million (decimal) words at each time 






Figure 1—Contours of constant equivalence ratio for gas jet injection. Fuel-rich contours are near the center and fuel-lean contours are near the perimeter of 

the jet. Air swirl rate is 4000 RPM. 
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step. If assumptions are made with respect to physical sym¬ 
metry, limiting the applicability of the model to somewhat 
idealized geometrical configurations, then the memory size 
requirements of the model are brought down to a more 
manageable 30,000-50,000 words, but some generality has 
been sacrificed. 

Another injection model often used 3,5 consists of treating 
the injection as a gaseous jet into a gaseous medium. By 
eliminating the droplet phase a substantial reduction in the 
computational requirements is possible. In many practical 
applications, particularly in diesel engine chambers where 
conditions are close to the thermodynamic critical point of 
the fuel, the gas jet approximation is quite realistic. We can 
illustrate this approach with results computed by one of us 
(L. H.) for the case of a gaseous fuel jet injected into a 
swirling air flow. The combustion chamber is a thin circular 
disk representing a typical diesel combustion chamber near 
TDC, the point of maximum compression. Fuel is injected 
into the air flow which is swirling at a rate of 4000 RPM. 
The fuel is emerging from the injector at a velocity of 500 
meters per second and is injected for 2 ms. As the gaseous 
fuel jet moves through the chamber, air and fuel gradually 
mix together. This mixing takes place preferentially at the 
edges of the gas jet where the shear force between the fuel 
and air is largest. In addition, the swirling air flow further 
enhances the fuel-air mixing. We can summarize the overall 
gas jet evolution by plotting contours of constant equiva¬ 
lence ratio <f> in the combustion chamber at a sequence of 
times. The equivalence ratio is defined as the ratio of actual 
fuel concentration at a point to the fuel concentration which 
would exactly consume all of the oxygen at that same point. 
Therefore for <£>1 the mixture is locally fuel rich, while for 
</><l the mixture has excess oxygen. In Figure 1, we plot 
contours of constant equivalence ratio for values of </>=2.0, 
1.5, 1.0, and 0.5 at a sequence of times of 1 msec, 2 msec, 
3 msec, and 5 msec after the beginning of injection. We see 
that the fuel jet travels across the chamber and gradually 
mixes with air. Large scale eddies are formed at each side 
of the jet due to air entrainment, and the swirling air flow 
deflects the jet in the direction of the swirl. These results 
have been compared and agree with experimental data. 3 The 
computer model has been used to examine the results of 
variations in a variety of quantities, including air swirl rate, 
injection velocity, and injector location. This type of param¬ 
eter variation in a computer model can be much more eco¬ 
nomical than a corresponding experimental program and can 
be used to indicate promising conditions for a much smaller 
number of experiments. This process reduces the time and 
expense of a research program. 

Once the fuel charge has been prepared, the mixture must 
then be ignited. The ignition process, either by compression 
as in the conventional diesel engine, or by spark discharge, 
presents many serious problems to a modeling effort. In the 
case of spark ignition, it is not yet clear how the electrical 
discharge initiates the combustion. It appears that, for a 
relatively short period of time, even the assumption of local 
thermodynamic equilibrium does not apply, i.e., that the 
translational, rotational, and vibrational temperatures of the 
gas are not all equal. There is an appreciable amount of 


ionization of the gas molecules, so the chemical kinetics 
properties of the gas are poorly understood. Also, fluid flow 
properties in the vicinity of a spark discharge are not always 
subsonic, and shock waves can be generated. In addition to 
these physical model inadequacies, ignition modeling pre¬ 
sents serious numerical problems. The characteristic phys¬ 
ical dimension of a spark region is very small, of the order 
of a millimeter or less. Yet the overall size of the combustion 
chamber is many times larger, indicating that lack of phys¬ 
ical resolution may be a problem in computations. The ig¬ 
nition process is a difficult one for numerical modeling and 
a great deal of work is needed in this area. 

Once the charge is ignited, a flame will propagate through 
the combustion chamber. The chemical kinetics of hydro¬ 
carbon fuel oxidation is another complex subject which is 
subject to both a lack of fundamental chemical information 
and by computational limitations. Detailed information on 
the reaction mechanism, intermediate chemical constituents, 
and elementary reaction rates is currently available for only 
the simplest of fuels. The chemical kinetics of methane 
(CH 4 ) involves as many as 25 chemical species and 50-100 
elementary chemical reactions in a reasonably complete de¬ 
scription of its oxidation and energy production. 6 Practical 
fuels such as gasoline involve many more constituents and 
reactions, many of which are unknown. As a result, simpli¬ 
fications are usually made in the expressions for the reaction 
mechanisms for almost all practical fuels. 

Reaction mechanisms, exact or simplified, can then be 
coupled with fluid flow equations in a spatially varying, time 
dependent numerical model of flame propagation. Many 
such models exist. In one example, a comparison was made 
between combustion in a homogeneous charge engine and 
in a stratified charge engine. 7,8 Computed results indicate 
that charge stratification, combined with increased compres¬ 
sion ratio, can make significant contributions towards re¬ 
ducing pollutant formation and improving fuel economy. 
Typical results are shown in Figure 2, indicating the reduc¬ 
tion in carbon monoxide and nitric oxide levels as a result 


STRATIFIED CHARGE FLAME PROPAGATION _ 



Figure 2—Total calculated concentrations of CH 4 (methane), NO, C0 2 , and 
CO for homogeneous charge (1) and stratified charge (2) combustion. 
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of charge stratification. In addition, preliminary calcula¬ 
tions 7,8 indicate that the stratified case may also provide 
increased fuel economy in terms of pressure increase per 
unit mass of fuel burned over the conventional homogeneous 
charge case. 

In the analysis of stratified charge engine combustion, 
experimental techniques are not yet able to obtain enough 
data to adequately evaluate many performance characteris¬ 
tics. Time- and space-resolved information inside an engine 
chamber, including temperature, chemical species concen¬ 
trations, and bulk fluid motion is not usually available, even 
with advanced laser diagnostic techniques. While this situ¬ 
ation will eventually be remedied, at present computer mod¬ 
eling offers the only access to much of this type of infor¬ 
mation. 

Another exceedingly important area of combustion re¬ 
search in automotive engines which is at present best ap¬ 
proached by modeling studies, is that of flame quenching. 
Flame quenching can occur in several physically distinct 
regimes. Perhaps the best known quenching phenomenon 
results from a flame encountering a relatively cold wall of 
the combustion chamber. The gas temperature drops sharply 
in a boundary layer to the temperature of the wall; that 
temperature close to the wall is too low to sustain oxidation 
reactions, so the fuel which is in this region remains un- 
bumed. During the subsequent exhaust phase of the engine 
cycle these unburned fuel regions are swept out of the cham¬ 
ber and are a major source of the unbumed hydrocarbon 
component of combustion pollutants. Modeling of this phe¬ 
nomenon is made difficult by two major factors. First, very 
fine physical resolution is needed in this type of problem, 
since the boundary layer has a thickness of less than a 
millimeter and this layer must be subdivided into a reason¬ 
able number of computational zones. Also, the mechanism 
of wall quenching almost certainly depends strongly on the 
details of the interaction between the thermal boundary layer 
and a number of elementary reaction rates, so this problem 
also requires a complete chemical kinetics model. Both the 
resolution and kinetics aspects of this problem require very 
large computers and sophisticated computational methods. 
Progress in modeling wall quenching processes is only now 
beginning to be made. 

A physically distinct type of quenching can occur in a 
reciprocating engine, particularly under conditions of lean 
combustion and late ignition, where the piston motion rap¬ 
idly increases the volume of the combustion chamber. This 
expansion lowers the reacting gas temperature and its den¬ 
sity. Since the rates of the great majority of elementary 
chemical reactions decrease rapidly as the temperature and 
density are reduced, this volume expansion can significantly 
reduce the rate of fuel oxidation, even to the point of totally 
halting the flame propagation. Laboratory experiments and 
an accompanying numerical modeling study have together 
been able to provide a fairly complete physical description 
of this type of bulk quenching of hydrocarbon oxidation by 
rapid expansion of the reaction gases. 9 

Another type of numerical modeling of combustion pro¬ 
cesses in automotive engines is shown by an example of a 
multidimensional fluid mechanics model. It is currently not 


feasible to include a detailed chemical kinetics model with 
two or three independent space dimensions. This limitation 
can best be illustrated by remembering that in a typical 
kinetics model, approximately 20-30 differential equations 
must be solved, one for each chemical constituent. In a 
spatially varying model, these species equations must be 
solved for each computational cell, plus the usual fluid me¬ 
chanics variables such as velocity and density. For a one¬ 
dimensional, time-dependent model with 100 cells this re¬ 
quires the solution of about 2500 coupled partial differential 
equations at each time step and takes up to an hour of CPU 
time on a CDC 7600 computer. For a computation with two 
space dimensions and 100 cells in each dimension, the same 
problem would consume over 100 hours, or four days of 
computer time, making such a computation practically un¬ 
feasible. As a result, almost all engine models which con¬ 
sider two space coordinates simplify the reaction model to 
one or two “global” reactions which are intended to give a 
reasonable representation of the chemical heat release which 
can be expected. Within this context, useful information 
about fluid flow properties which may not depend on the 
chemical kinetics details may be examined numerically. In 
an example of the use of this type of model, we compute 
the fluid mechanical motion of the gases in a motored engine 
chamber over a full four-stroke cycle. The engine is assumed 
to be axially symmetric, with a single intake-exhaust valve 
located on the chamber axis. The piston motion and valve 
timing represent an engine speed of 2500 RPM. 


Piston 


Figure 3—Initial grid for motored engine calculation. Only half of the grid 
is used for the calculation, it is reflected about the axis here for 
presentation purposes. 

























































Computer Modeling of Automotive Engine Combustion 


43 


Figure 4—Time 4ms. Maximum velocity 174 m/s. 


Figure 6—Time 12ms. Maximum velocity 70 m/s. 


In Figure 3, we show the initial grid used in this calcula¬ 
tion. The valve, head, piston, and cylinder walls are all 
represented as solid boundaries, with no heat conduction 
through them. The pressure at the left boundary is held 
constant at one bar, while the piston and valve move across 
the grid, which is stationary. This figure and the others in 
this series are taken from a sequence of computer-generated 
graphical output. This data has been put into the form of a 
continuous film which is a very useful and informative man¬ 
ner for presentation. Figures 4-15 are vector plots of the 
velocity for every other computational cell. For each plot 
the vectors are scaled linearly with the maximum velocity 
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Figure 7—Time 16ms. Maximum Velocity 47 m/s. 
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Figure 5—Time 8ms. Maximum velocity 178 m/s. 


Figure 8—Time 20ms. Maximum velocity 42 m/s. 
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Figure 9—Time 24ms. Maximum velocity 14 m/s. Figure 11—Time 32ms. Maximum velocity 10 m/s. 


always being the same length. The plots are at 4 msec in¬ 
tervals, which is equivalent to 60° increments of crank angle. 

The intake stroke is shown in Figures 4-6. At 4 msec a 
toroidal vortex is formed downstream from the valve and 
the flow velocity of the air through the valve has reached 
half the sound velocity of the air. This high velocity contin¬ 
ues through 8 msec, by which time two toroidal vortexes 
have formed. At 12 msec, which is the bottom of the stroke, 
the flow through the valve has stopped and the peak velocity 
is approximately one fifth the sound speed. The flow in the 
cylinder chamber has now broken up into a number of small 
vortexes and is quite complex. At 14.4 msec the intake valve 
closes and it and the manifold are removed from the com¬ 
putational problem. At 16 msec the compression stroke has 
begun and the flow continues to decay since it no longer has 
the flow through the valve to drive it. Figure 7 and Figure 



Figure 10—Time 28ms. Maximum velocity 12 m/s. 


8 show the piston approaching the top and at the top of the 
compression stroke. At this time the grid resolution is fairly 
coarse and so the definition of the flow is rather poor. This 
could be corrected either by rezoning the grid or by using 
a grid which moves with the piston. 

The expansion stroke is shown in Figures 9-11. On this 
stroke the gas motions caused by the intake valve have 
largely been dissipated, and subsequent gas motions are 
dominated by the motion of the piston. Through most of the 
expansion stroke the maximum velocity of the gas is close 
to the piston velocity. 

At 33.6 msec the manifold and exhaust valve are put back 
into the grid. Figure 12 shows the calculation at 36 msec, at 
the bottom of the expansion stroke. The inflow of gas 
through the valve and the small vortex in the cylinder cham¬ 
ber are caused by the fact that the exhaust valve opens 2.4 
msec before the end of the expansion stroke, so air can be 
drawn in from the manifold. Figures 13-15 show the exhaust 
stroke. The distinctive feature of this flow is that it is rela¬ 
tively simple. The flow reaches its maximum velocity at 44 



Figure 12—Time 36ms. Maximum velocity 48 m/s. 
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Figure 13—Time 40ms. Maximum velocity 113 m/s. 


msec. The velocity at this time is two thirds the sound 
velocity. 

These flow visualization calculations can consume signif¬ 
icant amounts of computer time, particularly if a fine grid is 
needed to resolve the flow. Thus it is important in this kind 
of calculation to take maximum advantage of computer ar¬ 
chitecture and to have efficient algorithms for solving the 
equations. For example, the program used for the flow vis¬ 
ualization calculation achieves a speed up of about a factor 
of two by using the parallel processing capabilities of the 
CDC-7600. Another factor of two was attained by using a 
more efficient algorithm for solving the equations. Thus a 
calculation which once took four hours of computer time 
can now be done in one hour. 

In the examples presented of current research in computer 
modeling of automobile engine combustion, we have em¬ 
phasized three major points. First, there are difficult theo¬ 
retical problems which remain to be solved before many 
combustion problems can be effectively approached. For 
these problems, such as liquid atomization and turbulence, 
even the physical and mathematical formulations are not yet 
satisfactory. Major efforts are under way, directed towards 
solving some of these theoretical problems, and considerable 
progress can be anticipated in the next few years. Second, 
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Figure 15—Time 48ms. Maximum velocity 46 m/s. 


efficient numerical solution methods are needed in many 
cases where the mathematical model is satisfactory, but the 
computational methods are inadequate. Like in the case of 
chemical kinetics rate equations, where the development of 
stiff equation software made efficient solutions possible, 
problems remain which are primarily numerical in nature. 
One of the most prominent examples of this need is in the 
treatment of two-dimensional and three-dimensional geo¬ 
metrical problems involving spatial transport. At present, 
nearly all of the available solution algorithms for these prob¬ 
lems are slow, unwieldy, and inaccurate, and a great deal of 
work needs to be done. The development of faster and larger 
computers, with new computer architectures, provides con¬ 
siderable promise in this area. Techniques which were not 
feasible on earlier computers may be attractive on vector 
pipeline or parallel processors. Finally, many additional lim¬ 
itations are placed on the productivity of computer combus¬ 
tion models by hardware limitations, particularly those of 
core size and computation speed. In this area as well, ad¬ 
vanced computer development should be of great benefit to 
combustion research. The magnitude of the theoretical and 
computational tasks involved in modeling of automobile en¬ 
gine combustion makes it certain that this research will 
continue to require the most advanced computers and nu¬ 
merical techniques available. The automotive engine ac¬ 
counts for a significant part of current energy usage and an 
even larger share of the petroleum usage. Even small im¬ 
provements in combustion properties can have a sizable 
impact in many areas, and computer modeling of automobile 
engine combustion has the capability to make significant 
contributions to improved combustion characteristics. 
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INTRODUCTION 

After years of steady, predictable model changes, the Amer¬ 
ican automobile industry is in the midst of the most intense 
product changeover in its history. By 1985, manufacturers 
must introduce new lines of vehicles which produce practi¬ 
cally no pollutants, and which consume much less fuel. One 
obvious step in the solution is to resize the automobile, 
thereby producing a lighter, smaller vehicle, without sacri¬ 
ficing interior roominess and passenger safety. To accom¬ 
plish the design of such a car, the structural engineer will 
need to use imaginative concepts and a variety of new ma¬ 
terials. The computer is and has been an indispensable tool 
in carrying out these changes. 

In the past, vehicle structural design was based on expe¬ 
rience, extensive laboratory testing, and finally, proving 
ground evaluation and development. Analytic methods 
though available were extremely difficult, if not impossible 
to apply to the complex automobile structure (Figure 1). 
Emphasis, therefore, was on experimental determination of 
structural behavior and performance. However, the de¬ 
mands on the automobile designer increased and changed 
rapidly, first to meet new safety requirements and later to 
reduce weight in order to satisfy fuel economy requirements. 
Experience could not be extended to new vehicle sizes and 
performance data was not available on the new criteria. 
Mathematical modeling was therefore a logical avenue to 
explore. First, computer simulations of specific structural 
behavior were developed, especially in the area of auto¬ 
mobile safety. Later the development of computer graphic 
packages allowed more accurate representation of automo¬ 
bile geometry with less engineering time required to generate 
the model. Most recently the finite element method, a com¬ 
puter dependent numerical technique, has opened up a 
whole new approach to vehicle structural design. 1 

The exponential increase in computerized analysis of au¬ 
tomobile structures is reflected in the growing number of 
Society of Automobile Engineers (SAE) structural analysis 
papers presented in recent years. Over the last ten years, 
this increase has been at a compounded rate of 25 percent 
per year. A further sign of recent interest in finite elements 
and other structural analysis methods in the automotive 


industry can be found in the proceedings of two international 
conferences on Vehicle Structural Mechanics held in 1974 
and 1977 by SAE. 2,3 The analytic methods discussed at these 
meetings made it possible to evaluate and optimize the au¬ 
tomobile structural design. 

Starting with the 1976 Cadillac Seville, all new General 
Motors automobiles have been evaluated and developed 
using the finite element method. Without these advanced 
computer aided analytic tools, it would not have been pos¬ 
sible to make the progress that resulted in substantial weight 
reduction and still retain occupant safety, comfort and in¬ 
terior space requirements. 


COMPUTER MODELS FOR VEHICLE SAFETY 

In the early sixties, barrier impact testing was the only 
method available for evaluating the crashworthiness of a 
vehicle structure. 4 Later in that decade computer simulation 
techniques, based on test data, were used to help understand 
the dynamics of collision of both the vehicle 5 and the oc¬ 
cupant. 6,7 In 1970 simulation was advanced to the point that 
the technique became a reliable design tool in the automotive 
industry. All these methods, however, required experimen¬ 
tal inputs to represent the large plastic deformations in the 
structure. In particular, Kamal and Lin 5,8 represented the 
major vehicle structural components (such as the frame, 
sheet metal, drive line, fire wall etc.) as nonlinear resistances 
(Figure 2) whose force-displacement behavior are obtained 
from static crush tests. The body, engine, and transmission 
cross member are represented by lumped masses. The equa¬ 
tions of motion are then solved in which the dynamic resis¬ 
tive forces of the crushing structures are each assumed to 
increase linearly with the instantaneous rate of deformation 
for that element of the structure. While these were important 
developments, there was still the need to solve the structural 
deformation problem analytically so that one could predic- 
tively assess the crashworthiness of a particular design. 
Given the irregular geometry of the automobile structure 
(Figure 1), the finite element method was a natural tool. 
Melosh and Kelly 9 outlined in 1967 the problems that would 
be encountered in applying the method to predict car crash 
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Figure 1—Typical automotive body structure 


response. This, however, has yet to be accomplished for a 
full vehicle. 

The structural problem to be solved is that of determining 
the dynamic forces and displacements for the automobile 
structure which is made up of shells, beams, corrugated 
panels, and irregular bars all subjected to large plastic de¬ 
formations. Many of the components are made of thin- 
walled beams and columns whose sections collapse during 
these large high-rate deformations. 

In the case of certain components or elements of an au¬ 
tomobile structure, progress has been made. Ni 10 has mod¬ 
eled the impact response of a curved box beam-column using 
a finite element-finite difference technique. This type of 
structure (Figure 3) is used in the car frame and underbody 
structure. Good agreement with experimental measurements 
was obtained (Figure 4) for both aluminum and steel frames 
of various dimensions. 

Chang 11 also discovered the importance of local defor¬ 
mation in his study of the automobile body structure. He 
presented an approximate technique for estimating the ulti¬ 
mate load-carrying capacity of the body structure without 
the need for experimental data (Figure 5). This procedure 
has been very useful in determining whether the body (or 
passenger compartment) will withstand the forces generated 
while the frontal structure crushes and decelerates the ve¬ 
hicle. It should be mentioned, however, that this is an ulti¬ 
mate load analysis quite different from what is needed in the 



Elastic Model Forestructure 

of Body Simulation 

Model 

Figure 2—Mathematical model for the analysis of vehicle frontal impact 



Figure 3—Dimension of curved box beam-column in mm (inches) 


frontal structure, namely the dynamic force-deformation 
function. Also metal panel deformations do not play as im¬ 
portant a role in this problem as they do in frontal crush. 

To summarize, computer simulation of vehicle impact is 
widely used to solve vehicle safety problems. Predictive 
analytic techniques are being developed, with progress being 
reported every year using numerical techniques heavily de¬ 
pendent on the digital computer. 

VEHICLE MODELS FOR VIBRATION 

Until the finite element method became available, dy¬ 
namic modeling was on the basis of gross models with only 
a few degrees of freedom. 12 However, mathematical mod¬ 
eling using substructures or building blocks had been well 
understood for some time. 13 What was needed for imple¬ 
mentation was the development of computers with large in- 
core storage capability to contain information on the large 
number of simultaneous equations or degrees of freedom 
required for a complete vehicle model. The final ingredient 
for success was the development of a “general purpose” 
structural computer code, which is user oriented, and has 
extensive documentation facilitating application by nonspe- 
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Figure 4—Comparisons of analysis and experiment for dynamic force- 
deflection relationship (steel, 15.4 mph and aluminum, 15.3 mph). 
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Figure 5—Finite element model of car body side structure 


cialist design engineers. The NASA STRuctural ANalysis 
(NASTRAN) program is one such code. 14 A part of this user 
convenience has been the ease of model visualization and 
results interpretation arising from the choice of structural 
displacements as the unknowns in the formulation of the 
equations of motion. 

Current methods 

Within the last five years, complete vehicle structural 
models based on the finite element method have become a 
practical alternative to testing. More or less complete finite 
element models were described as early as 1970, by Selna 
and Salinas. 15 The method has been widely used both in the 
United States and overseas. This approach enables the an¬ 
alyst to generate a model based on structural blueprint di¬ 
mensions and material properties, prior to any prototype 
build. Modeling is often done with enough detail so that the 
result may be used for examining local behavior as well as 
overall dynamic properties. The use of many thousand de¬ 
grees of freedom (equations of motion) is not unusual. A 
typical model for a unit body car is shown in Figure 6. A 
typical beaming vibration mode shape resulting from com¬ 
puter finite element analysis of this car is shown in Figure 
7. 

The cost of performing a vibration analysis on a complete 
several thousand degree-of-freedom model can be prohibi¬ 
tively expensive even on a modem high-speed computer. 
Usually, dynamic degrees of freedom are needed only at 
points of significant mass and moment of inertia. Conden¬ 
sation techniques can be used in this case to reduce the 
unknowns to a more manageable number. 16 

Substmcturing is also useful for cost and time savings. 13 



Figure 6—Model of unit body vehicle 




Figure 7—First beaming mode sprung mass model, 23.1 Hz 


The key to success is the reduction in the total number of 
degrees of freedom (dof) as the model becomes more spe¬ 
cific; that is, as we progress from a continuum (infinite dof), 
to a collection of finite elements (—2500 dof), to several 
substructures (—250 dof), to a structural system (—25 dof). 
This reduction procedure saves time and money in the com¬ 
putations without materially reducing our physical under¬ 
standing. Substructuring is a good illustration of how one 
can extend the capabilities of the computer when the naive 
temptation is to represent more and more details in the hope 
of increasing the fidelity of the model. 

Structural acoustics 

Another area receiving considerable recent interest for 
modeling and computer solutions is low frequency vibrations 
(up to 200 Hz) of the interior volume of the passenger com¬ 
partment. Finite element analysis is attractive for these sys¬ 
tems of complicated geometry. Considering first the case of 
free vibrations, the passenger compartment air volume is 
subdivided into the usual finite element mesh, as shown in 
Figure 8. For low frequencies (less than 100 Hz), the sound 
pressure field has very little cross-body variation, and a two- 
dimensional model suffices. 

The fluid mass and elasticity may be described relative to 
the finite element mesh as for a structural system. The 
calculations of natural frequencies and mode shapes then 
proceed in a straightforward manner. 17 The results are 
shown for four configurations of a station wagon in Figure 



Figure 8—Finite element model of station wagon passenger compartment 
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(d) WITH RIGID BAFFLE 

(+), (-) Relative Sign of Over (Under) Pressure 
— -- Nodal Line (Zero Change in Pressure) 

Figure 9—Acoustic mode shapes for four configurations of station wagon 


the General Motors Technical Center, the SMUG (Structural 
Modeling Using Graphics) Program 18 is based on earlier sys¬ 
tems designed for recording and manipulating styling line 
data. 19 This program allows the analyst to generate a finite 
element model automatically from styling data during an 
interactive console session. The program output is a struc¬ 
tural data file ready for input to NASTRAN. A typical plate 
model constructed using SMUG is shown in Figure 10. 

Computer graphics is useful in several other areas, in¬ 
cluding checking finite element models for errors, and gen¬ 
erating output plots of vehicle vibration modes. Graphics 
programs have also been developed for detecting interfer¬ 
ence among solids and surfaces, a prerequisite for allocating 
vehicle interior space efficiently. 20 


9. The inclusion of flexible walls in the model is also easy 
to accomplish. Once the resonant properties of the com¬ 
partment and surrounding structure are established, forced 
vibration analyses may be performed to predict the interior 
noise for various operating conditions. 

Computer graphics 

The magnitude of data input required for finite element 
models dictates the use of automated data generation and 
handling, usually using interactive computer graphics. At 



VEHICLE MODELS FOR RESIZING AND 

OPTIMIZATION 

Resource and fuel savings are directly related to resizing 
and optimization of vehicle components. The availability of 
the computerized structural analyses discussed in previous 
sections of this paper led to the ability of automotive de¬ 
signers to resize production vehicles quickly without sacri¬ 
ficing interior roominess and passenger safety. For example, 
1978 GM intermediate cars have been lightened by 310 kg 
(685 lb) on the average (Figure 11). The initial resizing re¬ 
sulted in a 68 kg (150 lb) savings. Subsequent compounding 
effects led to a total 15 percent mass reduction. These com¬ 
pounding effects result when weight reduction in one area 
permits reduction in another area. A good example is the 
bumper system. Clearly a narrower car needs a narrower 
bumper, saving 4 kg (9 lb) from resizing. Moreover, by 
reducing the total car mass through resizing of other com¬ 
ponents, the bumper requirements dropped, leading to an 
eventual overall savings of 30 kg (67 lb). 

The next step in this process of design improvement might 
be to computerize the selection of a “best” design based on 
numerical criteria. This optimization process is an area of 


WEIGHT SAVINGS FROM RESIZING = 150 Pounds Average 



RESIZED WIDTH RESIZED IEH6TH TOTAL 3AVIAS8 


S InchM = US pounds svsrsgo 17 Inctm * 82 pounds svsrsgs 150 pounds svsrago 


TOTAL WEIGHT SAVINGS FROM COMPOUNDING = 685 Pounds Average 
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current research interest, and has resulted in recent papers 
on ride quality enhancement 21 and optimized vehicle crush 
resistance. 22 This concept of design through analysis has 
also been applied to the construction of an experimental 
automobile structure, 23,24 producing results of use in guiding 
production vehicle designers. 

Material substitution studies 

Another aspect of car design which can produce large 
mass and energy savings is that of material substitution. 
Using the technology of computer structural analysis and 
given explicit design criteria, the designer can quantify his 
engineering judgment regarding the benefit of a material 
substitution. For example, the effect on bending and torsion 
vibrations produced by a change in the mass of a “non- 
structural” hood panel of aluminum, or a change in the 
stiffness of a frame rail can be determined quickly using the 
computer. The various types of structural requirements to 
be considered in a material substitution study are discussed 
in a recent paper by Chang and Justusson. 25 

Computer stress analysis 

In addition to the overall stiffness and dynamics computer 
models which have proved essential for vehicle resizing and 
material substitution studies, there has been considerable 
usage of finer mesh analysis models for computer stress 
analysis studies. Several papers representative of industry 
accomplishments in providing lighter, more efficient com¬ 
ponent designs appear in References 2 and 3. Problems dealt 
with include stress analysis of doors and other sheet metal 
parts, thermal stresses in brake drums, engine and trans¬ 
mission component design, stress analysis of a rear axle 
carrier, stress and deflection analyses of truck and tractor 
roll-over protection structures, calculation of gear stresses, 
stress and deflection analyses of (possibly nonlinear) sus¬ 
pension springs, and turbine and radial compressor com¬ 
ponent analysis. 


FRICTION AND DRAG REDUCTION 

Friction and drag reduction is another area of opportunity 
for the automotive engineer responsible for producing an 
energy and resource efficient vehicle. Because of the high 
degree of sophistication needed to model these phenomena, 
this area is one where the computer has already been of 
considerable use and has great future potential. This section 
will be divided into three parts, discussing aerodynamic 
drag, tires and lubrication. 

Aerodynamic drag 

Because of its timeliness and importance in fuel economy 
studies, this subject was the topic of a recent GM Research 


Laboratories symposium. 26 For current cars, the rolling re¬ 
sistance (due to the mechanical parts of the vehicle) and 
aerodynamic drag are about equal near 50 km/h (30 mph), 
but the aerodynamic losses are twice the mechanical losses 
at 90 km/h (55 mph). 27 As mechanical improvements occur, 
this trade-off speed will drop, emphasizing the aerodynamic 
losses more in the future. 

The potential for energy savings is substantial. A 40 per¬ 
cent reduction in aerodynamic drag is felt to be feasible by 
aerodynamicists, 27 and might produce a 15 percent increase 
in fuel economy. Although the opportunities for energy sav¬ 
ings are great in this area, direct computer assistance has 
been small to date due to the turbulent and usually unsteady 
nature of the three-dimensional flow field. Problems and 
prospects for analytical solutions in this area are discussed 
by Landahl 28 and by Hirt and Ramshaw. 29 At present, a 
finite difference technique has been applied to some prob¬ 
lems, with finite element methods offering a second ap¬ 
proach in the future. 


Tires 

Tire mechanics has been an active research area recently, 
and was the subject of a short course at the University of 
Akron in 1977. The variety of available numerical simulation 
techniques is discussed in one section of the course notes. 30 
More specifically, because tire rolling losses are an impor¬ 
tant energy dissipator at all speeds, the relationship between 
tire rolling losses and fuel economy was the subject of a 
recent SAE-DOT research and development planning work¬ 
shop. Discussion included the topics of numerical simulation 
of rolling tires and an analytical method for tire power loss 
calculations. 31 


Lubrication 

The refinement of modem lubrication theory is of impor¬ 
tance to the automotive engineer in reducing internal friction 
losses to the theoretical minimum. These losses occur not 
only in bearings, but also as friction between piston rings 
and cylinder wall (=25-50 percent of engine mechanical 
losses). Progress in this area has been impressive, in good 
measure due to the availability of large computers. Research 
on the film profiles which minimize the coefficient of friction 
and the total friction force for a given load in a slider bear¬ 
ing 32 are being extended to consider minimum friction loss 
in two-dimensional slider bearings, and in non-steady lubri¬ 
cation (i.e., piston rings sliding in an engine cylinder 33 ). 

Analysis of journal, thrust, and slider bearings has now 
been brought to the point of permitting a design engineer to 
correctly size such components interactively using computer 
aided design programs. For given geometry and material 
inputs, the computer immediately displays operating load, 
temperature, and power loss results. The designer can 
quickly refine the design to fit his needs. 34 
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SUMMARY 

Computer application in the automotive industry has had a 
rapid growth in the past 15 years. The world energy problem 
is but one of the challenges of recent times following safety 
and emissions. Computer models and methods are being 
used today to reduce weight, resize the automobile, reduce 
friction of moving parts, reduce aerodynamic drag, substi¬ 
tute materials, and assure product reliability and safety. This 
trend will continue and intensify in automotive engineering 
as progress is made in both computer hardware and the 
applied sciences. 
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The development and application 
of a mathematical model 
of enhanced oil recovery 

by HARVEY S. PRICE 

INTERCOMP Resource Development and Engineering, Inc. 

Houston, Texas 


INTRODUCTION 

A great deal of attention has recently been focused upon 
“enhanced oil recovery”; that is, the application of recently 
developed processes for recovering oil that would normally 
remain in a reservoir following depletion by conventional 
techniques. Since this heretofore unrecoverable remaining 
oil amounts to about 300 billion barrels in the U.S. alone 
(see Figure 1), one can see why enhanced recovery is getting 
so much attention. 

Among the broad class of enhanced oil recovery processes 
are processes called chemical flooding. These chemical 
flooding processes are extremely complex and really not 
completely understood. For example, a chemist in the lab¬ 
oratory can design a process which will recover 80 to 100 
percent of the oil-in-place from a representative piece of 
reservoir rock; however, when these same systems are ap¬ 
plied in the field, recoveries have ranged from 15 to 65 
percent. This implies that there are interactions which some¬ 
times occur in the reservoir that are sufficiently off design 
relative to laboratory work, to result in near or total process 
failure. To this technical uncertainty, add the problems of 
selecting a specific process for a given reservoir and the 
tremendous economic uncertainties that still exist; we see 
why chemical flooding is not projected to recover substantial 
amounts of oil for this country by the year 1985. 

One way to accelerate the amount of oil recovery from 
chemical flooding is to obtain a better understanding of what 
takes place in a chemical flooding project. This understand¬ 
ing can be achieved through the interaction between math¬ 
ematical models, laboratory results and actual field experi¬ 
ments. Much work is going on in the laboratory, and much 
work is going on in the field. However, until very recently 
there did not exist a mathematical model which could be 
used to better evaluate and understand the complex physics 
involved in all these processes and be used to scale the 
laboratory results to the field. INTERCOMP has developed 
such a chemical flooding model, and the development and 
application of this model is the subject of this paper. 


The term “chemical flooding” used here is only one of 
many names used by the oil industry to describe similar oil 
recovery processes. Other names include micellar-polymer 
flooding, low tension waterflooding and soluble oil flooding, 
as well as several trademarked names for processes devel¬ 
oped by specific companies. These have in common the 
injection of a chemical, normally a surfactant (or soap) to 
reduce the forces that trap oil in the rock in the presence of 
water. With these forces reduced, the oil can be mobilized 
and driven to a production well. Although the process is 
simple in concept, the specific interactions of the fluids 
involved are quite complex. 

In order to model the behavior of chemical flooding, a 
number of primitive unknowns need to be identified and the 
complex partial differential equations which describe mass 
balances on these unknowns need to be solved in one, two 
and three dimensions. This, however, is not all of the prob¬ 
lem. Because of the important physical processes which are 
taking place, the mathematical solution of these equations 
needs to be far more accurate in some instances than in any 
other petroleum recovery technique which is currently being 
routinely modeled. This paper presents results of some ap¬ 
proximations which have been successfully used by INTER¬ 
COMP to generate solutions with acceptable accuracy. 

This model is currently being used in an application to 
help engineer a micellar-polymer flood in the Bell Creek 
field for Gary Operating Company. The model has been 
effectively used to characterize two quite different chemical 
flooding processes and has aided Gary in selecting the proc¬ 
ess which will be used in the field. A discussion of the 
selection process and some preliminary engineering results 
which should enhance the performance of Gary’s field pilot 
are also presented. While the discussion of the Bell Creek 
Project is very preliminary, we believe that this is the be¬ 
ginning of an understanding which will ultimately lead to 
much better agreement between project design and field 
performance. If this goal is achieved, we will be well on our 
way to knowing how many of the 300 billion barrels of now 
unrecoverable oil can be recovered economically. 
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SPECIFIC TREATMENT 

Chemical flooding is the name we shall use to denote 
these reservoir processes characterized by the injection of 
an agent to reduce the surface tension between oil and water 
and, hence, allow the displacement and recovery of oil that 
is normally trapped by capillary forces as a waterflood re¬ 
sidual. The process is known alternatively throughout the 
industry as miscible-type waterflooding, 1 micellar flooding, 2,3 
low tension waterflooding, 4 surfactant flooding, 5-7 and sol¬ 
uble oil flooding. 8 

Although chemical flooding could be applied for second¬ 
ary as well as a tertiary oil recovery, the typical chemical 
flooding candidate is currently a watered-out oil zone; the 
in situ water being a normal oil field brine, high in total 
dissolved solids and divalent cations. The chemical flooding 
process thus consists of (1) a brine preflush to condition the 
formation and provide a controlled fluid environment that 
will allow optimum activity of the following surfactant sys¬ 
tem; (2) a small slug of a surfactant fluid, generally consisting 
of a dilute concentration of petroleum sulfonate in brine, 
with the possible additions of cosurfactants, polymers and 
other chemicals to stabilize the system, enhance the surfac¬ 
tant activity, reduce adsorption losses and control slug mo¬ 
bility; (3) a mobility control slug consisting of a dilute so¬ 
lution of polymer in brine, used to protect against backside 
dilution or overrunning of the surfactant slug by drivewater 
and to enhance areal and vertical sweep efficiency; and (4) 
a waterdrive to sweep the displaced oil, water and injected 
fluids to the producers. 

As pointed out by Knight et al., 15 use of a brine preflush 
is not universal and certainly there are nuances that are 
unique to each company’s process. However, the above 
description appears to be the generally accepted character¬ 
ization of the chemical flooding process. 

The fluid characteristics and transport mechanisms im¬ 
portant to the chemical flooding process have been studied 
extensively throughout the industry. A few of these studies 


are described in References 1-14 and we have included in 
our model the mechanisms and fluid properties indicated to 
be significant. 

The chemical flooding system can be represented by six 
species or components flowing in a maximum of two phases. 
We may think of species one through six as water, oil, 
polymer, surfactant, salt and divalent ions, and denote the 
phases as aqueous and hydrocarbon. However, if we write 
the conservation equations for species three through six as 
to allow for (a) partitioning between aqueous and hydrocar¬ 
bon phases, (b) adsorption on the reservoir rock, and (c) 
bulk transport and dispersion in both phases, other systems 
may be examined as well. In fact, the formulation used 
should allow the study of all of the mechanisms which have 
been indicated by the literature to be important. 

Model formulation 

The chemical flooding model solves for six components 
(or species) in two fluid phases. All six components may 
partition amongst the aqueous and hydrocarbon phases. 
Four components may disperse by both molecular diffusion 
and mechanical dispersion in both the longitudinal and trans¬ 
verse direction and in both phases. The same four compo¬ 
nents may adsorb on the reservoir rock, satisfying equilib¬ 
rium sorption isotherms. 

The primitive unknowns to be solved for by the simulator 
are: 

WA1 Mass fraction of water in the aqueous phase 

WH2 Mass fraction of oil in the hydrocarbon phase 

WA3 Mass fraction of component three (normally poly¬ 
mer) in the aqueous phase 

WA4 Mass fraction of component four (normally surfac¬ 
tant) in the aqueous phase 

WA5 Mass fraction of component five (normally mono¬ 
valent salt) in the aqueous phase 

WA6 Mass fraction of component six (normally divalent 
salt) in the phase saturation 

SA Aqueous phase saturation 

p Aqueous phase pressure 

The equations to be solved by the simulator are: 

Component 1 (Water) 

|^>S a PaWAl+</>S h p h SHl) + (1) 

V( ft WAlU a +p h WHlU h )-q 1 =0 

Component 2 (Oil) 

((/>S a p a WA2+ <£S h phWH2) (2) 

+V •( p a WA2U a + Ph WH2U h ) - q,=0 

Component 3 (Polymer) 

!^(<£pS a p a WA3+ 0pS h phWH3+ </>p. 3 ) 

+ V-(paWA3U a + p h WH3U h )-V-(^ p S a p a Ka 3 VWA3 
+ 4ShPhSh 3 VWH3)-q 3 =0 

( 3 ) 
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Components 4 (surfactant), 5 (monovalent salt) and 6 (di¬ 
valent salt) equations are the same as that for Component 
3, except that </> p is replaced by 4>. 

There are three constraint equations for the system. The 
first requires the sum of saturations be equal to one. This 
equation is satisfied by setting S h = 1-S a in a six component 
balance equations. The other two constraint equations re¬ 
quire that the sum of the mass fractions in each phase equal 
one. These equations are left explicitly in the formation as 
Equations (7) and (8). By ordering the unknowns as shown 
on the list of primitive unknowns, Equations (7) and (8) 
eventually become the “saturation” and “pressure” equa¬ 
tions. 

Assuming an isothermal system with compositions inde¬ 
pendent of pressure, we have the following equilibrium re¬ 
lationships for partitioning of species one through six be¬ 
tween the aqueous and hydrocarbon phases and for 
adsorption of species three through six on the reservoir 
rock: 


WHi=fi(WAj) j= i_ 6 for i= 1 through 6 

(9) 

PRi=fRi(WAj) j=1 - 6 for i=3 through 6 

(10) 

For equations of state, we have 

Pa=P a (WAi,p a ) 1=1 _ 6 

(ID 

Ph—PhfWHijPhji^-e 

(12) 

U a = - ( V Pa — P ae g VD) 

(13) 

Uh=— — (Vph-PhegVD) 

ph 

(14) 

where 

k a =k a (S a ,WAi) 1=1 _ 6 

(15) 

k h =k h (S a ,WAi,U a ) i=1 _ 6 

(16) 

Pa Pa(WAi,S a ,Ua)i=l-6 

(17) 

Ph=Ph(WAi,S a ) i=1 _ 6 

(18) 


Assuming that the capillary pressure is a function of both 
saturation and composition, we have finally 

Ph=Pa+P c (S a ,WAi) i=1 _ 6 (19) 


MODEL SOLUTION 

As if developing numerical solutions to Equations (1) 
through (19) is not complex enough, the numerical solution 
needs to be extremely accurate in order to be able to model 
the phenomena of dispersion and fingering. If it weren’t for 
these special problems, the numerical solution to Equations 
(1) through (19) could follow any normal finite difference or 
finite element type approximation, and for this reason will 
not be addressed in this paper. We will, however, address 
our attention at the specific problems of dispersion and 
fingering since these are truly important physical phenomena 
to be modeled and help to make some of the interesting 
points of this paper. 


ONE-DIMENSIONAL MISCIBLE DISPLACEMENT 
5% SLUG WITH M=l 



Figure 2 


Figure 2 is included here to illustrate the effects of nu¬ 
merical dispersion. This figure displays the results of a sim¬ 
ple, one-dimensional calculation. If the solution were infi¬ 
nitely accurate, the high spike shown on this figure would 
result from the effects of actual physical dispersion on the 
solution. As you can also see on Figure 2, the very smeared 
curve displays the effects of large numerical truncation error 
which has the exact same effect as a large physical disper¬ 
sion. Figure 3 is an illustration depicting the phenomenon of 
viscous fingering. This occurs whenever a more mobile fluid 
is injected into a less mobile fluid; the more mobile fluid 
wants to finger through the less mobile fluid to the producing 
well providing a thief zone for the injected fluid. This phe¬ 
nomenon results in cycling of the injected fluid without 
really displacing the fluid in place. In chemical flooding, 
because the costs of the chemicals injected are so extremely 
high, all of the processes are designed around putting in as 
little chemical in the ground as is absolutely necessary. 



Figure 3 
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HERMITE PIECEWISE CUBIC ELEMENTS 

Figure 4 


Because of this, the effects of both physical dispersion and 
viscous fingering are extremely important in the field and 
need to be modeled accurately if realistic results are to be 
obtained. 

There are two approaches to solving partial differential 
equations describing the chemical flooding process and still 


be able to achieve the kind of accuracies required in order 
to model dispersion and fingering. The first approach is to 
use high order finite element approximations and the second 
approach is to use some local grid modification such that 
the accuracy can be concentrated in the areas where it is 
required and coarse grids can be used elsewhere. The high 
order finite element approximations are depicted by Figure 
4 where I have illustrated here third order, cubic Hermite 
polynomials which can very effectively be used to generate 
fourth order accurate approximations. Figures 5, 6 and 7 
illustrate a specific type of local grid refinement. Here we 
have an original triangle grid in Figure 5 and two levels of 
refinement shown right at the injection well which is in the 
lower left-hand corner of Figure 6. Figure 7 illustrates a later 
point in time when this refinement has moved in somewhat 
away from the injection well in order to accurately describe 
a front which is moving away from the injection well. This 
is a local refinement that needs to be time varying and move 
along with the front. 

To illustrate the importance of these phenomena, I’ve 
shown on Figure 8 the effects of dispersion on recovery. 
You will note that as the physical dispersion is reduced, the 
peak concentration of the injected slug is increased and, 
therefore, the amount of oil recovered by continuous dis¬ 
placement of the injected fluid is significantly increased. 
Figure 9 is an illustration of a computer plot which shows 
a first order approximation to the injection of a slug of 
displacing fluid at a one-to-one mobility ratio. You will note 
how dispersed this particular slug is relative to Figure 10, 
which represents a fourth order correct solution to this prob¬ 
lem. You will note also Figure 11 where the use of local grid 
refinement has been used how much sharper and higher the 
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LOCAL REFINEMENT 

Figure 6 
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PEAK CONCENTRATION OF A 5% SLUG 



t(OAYS) 

Figure 8 


peak slug concentrations are than in Figure 9. Figure 12 is 
an illustration of a fourth order finite element approxima¬ 
tion’s ability to actually model viscous fingering in an un¬ 
favorable mobility ratio displacement and Figure 13 is an 
illustration of how the model is able to accurately predict 
recoveries at low dispersion levels for unfavorable mobility 
ratio displacements by introducing nominal heterogeneities 
into the system to propagate viscous fingers. 

One further phenomenon which results when finite differ¬ 
ence approximations are used to model unfavorable mobility 
ratio displacements is that of grid orientation. Figure 14 



Figure 9 
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Figure 10 


shows a repeated five-spot pattern and illustrates the two 
different grids that could be used to represent the flow of 
fluids in a five-spot. You will note that if the regions are 
totally homogeneous and if all the injection wells have the 
same rates and all the producing wells have the same rates, 
each of these repeated patterns should perform identical to 

5% slug, M-l 

Local Grid Refinement 
Using Triangular Elements 
t - 1280 days - 0.64 PV injected 



Figure 11 
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Figure 12 
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Figure 13 
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Figure 14 


every other repeated pattern. You will note from this figure 
that one of the grids is parallel to the coordinate axes while 
the other grid is running at an angle of 45° to the coordinate 
axes. These two grids are called parallel and diagonal for 
the sake of discussion. You will note in Figure 15 how the 
recovery performance for a very simple displacement proc¬ 
ess can vary depending on whether one uses the parallel or 
the diagonal grid. As mentioned before, these results should 
be identical and yet there is as much as a 20 percent differ¬ 
ence in the recovery of oil after one pore volume of fluid 
injected. This is clearly an intolerable error and needs to be 
eliminated. It can also be seen on this same curve that high 
order finite element approximations do not exhibit this grid 
orientation phenomenon. In fact, none of the finite element 
approximations exhibit grid orientation effects which plague 
finite difference schemes. 

We have illustrated in this section how the problem of 
describing a physical process such as chemical flooding re¬ 
quires numerical approximations which not only solve com¬ 
plex partial differential equation with complex nonlineari¬ 
ties, but which also must describe the important physics of 
the problem. Finite element approximations are definitely 
recommended when high order accuracy to model effects 
such as physical dispersion and/or viscous fingering are re¬ 
quired. 


MODEL APPLICATION 


FIVE SPOT RUNS WITH M-41 



Once a complex physical model, such as the one described 
by Equations (1) through (19), has been coded and debugged 
and running smoothly on a large-scale digital computer, one 
of the first things that needs to be done is to validate the 
model against actual physical processes in order to deter¬ 
mine that the model does in fact describe the complex phys¬ 
ics that it was intended to describe. This is really the stage 
of model development in which INTERCOMP is currently 
engaged; however, we have modeled two complex chemical 
flooding schemes and will illustrate here how the model was 
utilized in these particular instances. 

The actual study that was performed was to model two 
competing design philosophies and to utilize the model to 
determine which of these should be implemented in Gary 
Operating Company’s Bell Creek micellar-polymer pilot. 
The two chemical design philosophies which were modeled 
were (1) a small slug high concentration “soluble oil” sys¬ 
tem, and (2) a large slug dilute concentration, “optimal sal¬ 
inity” design. 

A number of laboratory experiments were run in order to 
obtain basic data for the model as well as to test the design 
philosophies in displacing oil from the Brea core. While a 
large number of experiments were actually modeled, I have 
chosen just two here to illustrate the actual capabilities of 
the chemical flood program. You will note from Figure 16 
how good an actual match was obtained for a .09 pore 
volume slug using the high water content “HWC” process 
and from Figure 17 we have illustrated the match for the 
soluble oil process using a .03 pore volume slug. The inter¬ 
esting features about these history matches were that both 
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HWC-0.09 P.V. Slug 



model runs utilized basically the same data and that only the 
different description of the physics resulted in the different 
model predictions. While these history matches are not per¬ 
fect, they could easily have been further adjusted and fine 
tuned. However, it was not felt that this was necessary since 
the purpose here was really to demonstrate the model’s 
capability to describe the physics of the two completely 
different systems. Once the history match was obtained then 
the model was used to predict the ultimate recovery from 
the cores using various weight percent surfactant concen¬ 
trations. You see from Figure 18 that here again the model 
did an excellent job of reproducing laboratory experiments 
and that clearly the soluble oil system, at least in the core, 
was significantly better than the high water content process. 

Once this history match was obtained the model was then 
utilized to make both areal and vertical cross section runs 
to see how these processes might deteriorate in the field. It 


SO - 0.03 P.V. Slug 



COMPARISON OF EXPERIMENTAL AND 
SIMULATED OIL RECOVERY 



is interesting to note that once we made these runs and 
combined the areal and the vertical runs to estimate what 
would be recovered from the pilot that the model predicted 
a 55 percent recovery with a soluble oil process and only a 
35 percent recovery with a high water content process. 
These results are still preliminary since we did not at this 
point look at any degradation due to layering for these sys¬ 
tems. However, it is very interesting to note that a process 
which recovers almost 90 percent of the oil in a core would 
be predicted to recover only 55 percent of the oil in an 
idealized pattern. This is far more like what is actually being 
observed in the field and is extremely encouraging that we 
have now reached the point where mathematical models can 
be used to predict oil recovery by chemical flooding. 

Again while this work is still very preliminary, I would 
like to point out one other very interesting thing that has 
been learned to date from this particular study. The field 
pilot has been designed as a five-spot with this expanded to 
the full field originally if the pilot is successful. However 
one of the interesting things that was noted in this work is 
that because of the diverging nature of the flow in a five- 
spot as the slug moves away from the injection well very 
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little oil is displaced around the edges of the pattern. In fact, 
almost no oil is being mobilized around the edges and most 
of the oil being recovered from the pilot is coming right out 
of the stream tubes that run between the injector and the 
producer. This gives the strong indication that line drives or 
patterns other than five-spots might be considerably better 
for recovering oil via chemical flooding. This work will be 
expanded upon and may in fact even result in some design 
modification for this very exciting and interesting pilot proj¬ 
ect. 


CONCLUSIONS 

In this paper we have presented some of the interesting 
aspects in the development and application of a chemical 
flood model. This clearly illustrates how computers and 
mathematical models can play an extremely important role 
in aiding the oil industry in recovering additional oil. The 
complex model which was developed is now being utilized 
to better understand laboratory work which will in turn 
enable us to scale the very important laboratory experiments 
to actual field scale projects and hopefully enable us to 
obtain a better understanding why some processes seem to 
work and other processes seem to fail. Once a process is 
characterized and the reservoir is characterized we feel that 
a mathematical model of this sort is an excellent predictive 
tool and should go a long way toward enabling the oil in¬ 
dustry to recover a substantial fraction of this 300 billion 
barrels of oil which is left in the ground and currently called 
unrecoverable. 
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Geophysical DP requirements could exceed 
the world’s GP capacity by 1985 

by CARL H. SAVIT 

Western Geophysical Company of America 
Houston, Texas 


INTRODUCTION 

The reflection seismic method of underground mapping is 
today the only practicable means of selecting a place to drill 
a new oil or gas well. As a consequence, the free world 
spends about two billion dollars a year for reflection seismic 
exploration. 

All of this activity consists of data gathering, data reduc¬ 
tion, and data analysis. In 1977 the industry recorded be¬ 
tween 10 14 and 10 15 bits of primary data in digital form. All 
of these data was processed at least once. Indeed, it is 
probable that the geophysics industry processes more pri¬ 
mary digital data than does any other activity. Worldwide, 
the industry dedicates computer power equivalent to a few 
dozen IBM 370/168’s to processing seismic reflection data. 

During 1977 new field techniques were introduced to im¬ 
prove the resolution of the entire process. These new tech¬ 
niques begin by recording about 20 times as much data per 
unit of survey as are being recorded in conventional, pres¬ 
ent-day practice. A further increase to about 50 times the 
present data rates will be needed to make the horizontal 
resolution of the system reasonably consistent with the ver¬ 
tical. Ultimately, algorithms that take full advantage of the 
increased recording density will require 1000 or more addi¬ 
tional computation steps per input point. Consequently, we 
perceive a need for a 50-fold increase in internal memory 
and a 50,000-fold increase in arithmetic speed. In terms of 
“power” (where power is defined as the product of internal 
memory size and the number of operations per unit time), 
these requirements translate to a 3xl0 6 -fold increase. 

It has been the experience of the past three decades that 
available computer power (as we have defined it) increases 
about one order of magnitude every 2.7 years. If, as seems 
reasonable today, this trend continues until 1985, our com¬ 
puters will then be about 10 3 times as powerful as they were 
in 1977. Also, as seems reasonable today, reflection seis¬ 
mologists will by 1985 be ready for the 3xl0 6 -fold increase 
we have projected. To accomplish the same number of units 
of survey in 1985 as were done in 1977, we could be facing 
a requirement for 3000 times as many 1985 model computers 
as the number of 1977 model machines we used in 1977. It 
would, in addition, be reasonable to expect the market for 
surveys to have doubled in the next seven years so that we 


may need to increase our demand by an additional factor of 
two. (Figure 1) 


DATA AND ALGORITHMS 

To understand the nature of the geophysical computation 
requirement, an overview of the basic physics of reflection 
seismology will be of some help. 

In reflection seismology a mechanical excitation is applied 
at or near the surface of the earth. Pick-ups also on or near 
the surface and relatively near the point of excitation detect 
the response of the earth to the excitation. From the com¬ 
plex pattern of energy returned to the surface, the explora¬ 
tion geophysicist attempts to select the longitudinal elastic 
waves (i.e., the sound waves) that have returned to the 
surface by reflection from underground interfaces between 
different rock layers. In the current state of the art, energy 
of all other kinds is rejected or suppressed as unwanted 
“noise.” 

Nearly all processing methods in general use into the first 
few years of this decade were based on the concepts of 
geometrical optics. In other words, sound waves were as¬ 
sumed to travel along rays and to be reflected or refracted 
at the point of intersection of each ray with an interface or 
discontinuity. (Figure 2) 

As a consequence of these assumptions, deriving a rep¬ 
resentation of the configuration of underground layering 
from the received acoustic signals involves the use of little 
more than the algebraic and trigonometric equations of an¬ 
alytic geometry. One major intermediate problem, that of 
deriving the velocity of sound as a function of depth from 
the surface, appears to require the solution of an integral 
equation. Instead, however, of attempting an analytic solu¬ 
tion in the face of noisy and incomplete data, the exploration 
seismologist has opted for the method of exhaustive search. 
He solves the forward problem of determining a suite of 
likely reflection times for selected parts of the data over an 
anticipated range of assumed velocities. The algebra re¬ 
quired is primarily the Pythogorean equation followed by a 
vast number of correlation calculations to determine the 
degree to which reflection times derived from the searched 
velocities fit the observed signals. 
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YEAR 


Figure 1—The power of typical commercially available computers has been 
growing about one order of magnitude every 2.7 years and should continue 
to do so through the 1980’s. Requirements of the seismic exploration industry 
have been growing and should continue to grow at a substantially faster rate. 

The necessary ray-path calculations are quite simple and 
so produce a relatively small computational load. Most of 
the computation time is devoted to the extraction of good 
acoustic “signals” from noisy data. The two approaches 
used to improve signal-to-noise ratios are the averaging of 
redundant data and filtering by means of a battery of so¬ 
phisticated techniques. 

By far the largest computing load stems directly from the 
high redundancy of data being taken. A typical field record 
of the response of the earth to one mechanical input consists 
of 3000 data values taken simultaneously at each of 48 
equally spaced surface locations. Each input thus produces 
about 150,000 floating-point numbers. Floating point is re¬ 
quired because the signal immediately after the instant of 
input can be four orders of magnitude larger than the signal 
six seconds later when the recording is finished. 

Typically one signal input is applied every 30 to 60 meters 
along a line of survey so that a seismic survey produces 



Figure 2—The geometric-optic model of reflection seismology postulates a 
unique raypath connecting an impulse, a given reflection, and an individual 
detector; and the angle of incidence is equal to the angle of reflection. 


some 2.5xlO 6 to 5.0xl0 6 floating-point data values per kil¬ 
ometer. 

Such surveys produce data redundancies ranging from 12 
to 24 or 48. When more redundancy is required to achieve 
useful anti-noise discrimination, more data are taken per 
kilometer of survey. Redundancies of 96 are not uncommon 
in special situations. 

In nearly half of all surveys on land, the input consists of 
a swept-frequency (chirp) signal of substantial duration. Sys¬ 
tems using this form of input require the recording of thirty 
times more data values per kilometer. Simple computation 
in the field compresses these data about 16-fold, but the 
result of this compression must be further processed before 
it reaches the computational level of unprocessed impulse 
data. 

Before redundant data can be combined or averaged, sev¬ 
eral distinct data processing steps must be applied. Those 
redundant data that apply to, or represent, the same sub¬ 
surface point are gathered from diverse input and pick-up 
points. The different sound waves reflected from the same 
subsurface point have travelled paths of different length and 
have encountered different inhomogeneities in their paths. 
Field recordings are, of necessity, made in real time so that 
data from different pick-ups must be recorded in time-mul¬ 
tiplexed order. 

Normally, the first computational step is to demultiplex 
the data into time-sequential value sets for each pick-up. 
Next, data are normalized both as to amplitude and time 
scale to approximate values that would have been produced 
by horizontal plane waves. The necessary transformations 
are non-linear but have simple geometric bases. 

In most cases the first of several filtering steps is applied 
to the data after the amplitude normalization and before 
application of the time-scale normalization. Almost always 
the filter is in the nature of a deconvolution; that is, an 
attempt is made to undo the various spread functions that 
characterize the effective shape of the input signal. That the 
input function has substantial duration is the result both of 
the imperfect nature of the mechanical impulse itself and of 
the reverberatory character of the earth materials near the 
input source. 

Designing the filter operators to be applied to the data to 
convert the input signal to a band-limited, simple impulse 
function represents a substantial computation load. While it 
is a relatively simple matter to use the Wiener-Levinson 
algorithm to invert a known (i.e., recorded-in-the-field) me¬ 
chanical impulse shape to a desired simple impulse, it is 
somewhat more burdensome to derive the analogous oper¬ 
ator to compensate for an unknown reverberation function. 
The usual method is to derive the amplitude spectrum of the 
reverberation function from autocorrelations of the unfil¬ 
tered data and to assume (on physical grounds) that the 
function is minimum-phase. Other more complex schemes 
are in use to various extents. 

All of the steps described to this point are necessarily 
applied to essentially all of the field-recorded data and, 
hence, represent the great bulk of the computation load. In 
those cases in which the velocity of seismic waves varies 
rapidly along the line of survey, the intermediate step of 
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velocity determination, described earlier, can dominate the 
computation requirement. 

In the last computer pass of these preliminary computa¬ 
tion steps, the data are combined or averaged to produce 
the compacted, signal-enhanced data set for final processing. 
Essentially all of the redundancy has now been removed so 
that data volume is reduced by the original redundancy 
factor of 12 to 96. Furthermore, the dynamic range of the 
data has been sufficiently reduced that results may validly 
be written as 16-bit, fixed-point numbers. 

Until quite recently, virtually all seismic reflection data 
were subjected to at most one or two additional filtering 
steps and were amplitude-conditioned in one of two alter¬ 
native modes and formatted for presentation in graphical 
form as hard copy. 

Such additional calculations as have been made within the 
geometric-optics framework have ranged from simple 
smoothing operations to elaborate ray-tracing routines but 
in the aggregate have had negligible impact on computer 
requirements. 

ACOUSTIC WAVE EQUATION 

We in the exploration-geophysics community have long 
been aware that the geometric-optics model of the seismic 
reflection process is, at best, a first approximation to reality. 
Even the fundamental assumption of the geometrical optics 
model—that the longest wavelengths involved are small 
compared to the dimensions of interacting elements in the 
medium—is at best weakly valid. Some of the wavelengths 
we use are indeed substantially greater than the dimensions 
of subsurface features of interest. 

The simplest model that enables us to relax the dimen¬ 
sional assumption required in geometric optics is acoustical. 
In the acoustical model it is assumed that the propagation 
of seismic energy can be described by the acoustic wave 
equation 


That equation is the mathematical expression of a physical 
model in which energy is propagated in a lossless medium 
solely by means of variations of pressure. 

It is well established that many of the defects of the ray 
propagation assumption of the geometric-optic model are 
mitigated or effectively eliminated by use of the wave-equa¬ 
tion model. Unfortunately, these advantages are gained at 
the cost of orders of magnitude increases in the required 
computations. 

This increase in computational load is inherent in the 
difference in the physical models. In the geometric-optic 
model, the reflection from a point on an interface obeys the 
specular rule—the angle of incidence is equal to the angle 
of reflection. Thus, an impulse from a particular surface 
point of origin can reflect at a given point on the reflector 
to only one receiver point on the surface. To process the 
information relevant to one subsurface point, it is only nec¬ 
essary to treat a number of data values equal to the design 
redundancy of the system. (Figure 2) 



Figure 3—The acoustic model of reflection seismology postulates that energy 
is radiated in all directions from every point on a reflecting surface. Accord¬ 
ingly, every detector receives some energy from each reflecting point in 
response to a given impulse. 


In the acoustic model, however, the reflection point is a 
new source and retransmits energy in all directions. Hence, 
every receiver records energy “reflected” from every point 
on the reflector in response to an impulse. Conceptually, we 
should use all data from the entire survey to determine the 
characteristics of each reflection point. While the details 
differ, operationally the process is analogous to the creation 
of an image corresponding to a hologram. (Figure 3) 

Obviously, an immediate and complete shift from the ray- 
theory to the wave-equation approach would have required 
more computer power than is available in the entire world. 
In any event, the necessary analytical foundations for such 
a change have not yet been developed. 

What has happened is that a working mix of the two 
models has been evolving. In its earliest manifestation the 
mix consisted of ray-path processing until a final com¬ 
pressed, enhanced data set is produced, following by a sin¬ 
gle-application of a limited-parameter version of the Kirch- 
hoff-integral solution to the wave equation. The number 
crunching required for this process when applied to 48-fold- 
compressed data was of the same order of magnitude as that 
needed for all the preceding steps. 

In today’s world, a battery of programming techniques 
and analytical simplifications have made the wave-equation 
approach more economical and the need for better data has 
made funding increasingly available. 

In the most sophisticated approach known to me to be in 
commercial use, the data are processed at least part way in 
the geometric-optic manner to derive model parameters 
which are then used for extensive acoustic-wave processing. 
The result of that processing is then further modified by a 
technique derived from geometric optics to compensate for 
an inadequacy in the wave-equation formulations available 
today. (Figure 4) 

Another few years should see a fully mature acoustic- 
wave technique to carry the entire processing sequence, but 
it will almost certainly require ray-path processing as a pre¬ 
liminary step to determine initializing or first-approximation 
parameters. Much of the processing may be iterative. 

For those who are inclined to worry about a possible 
static future, I would point out that the introduction of 
attenuation into the equations should suffice to keep every¬ 
one busy for some years beyond these projections. After 
that we could profitably spend a generation or two applying 
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Figure 4 —This cross section from offshore California represents a horizontal 
distance of about 26 kilometers and a total vertical dimension of 2.7 kilome¬ 
ters. About 3x 10* bits of data were recorded at sea. Processing through the 
compaction stage was performed in accordance with the raypath model. The 
compacted data were treated by acoustic wave methods, and the results were 
then further modified by another ray-theory-derived process. Processing costs 
were something over $6000. 


the tensor equations that describe the elastic behavior of 
real solids. 


HARDWARE REQUIREMENTS 

Since the seismic reflection industry went digital in the 
mid 60’s, it has been pressing hard at the frontiers of com¬ 
puter technology. The first group to undertake regular com¬ 
puter processing of seismic data designed and built a special 
purpose system because of the limitations of the then-avail- 
able GP systems. Seismic industry requirements soon stim¬ 
ulated the development by computer manufacturers of spe¬ 
cialized hardware to cut computing costs for some heavily 
used algorithms and thus bring overall costs to an acceptable 
level. In the initial years, correlation and convolution op¬ 
erations were the pacing items making up computer costs. 
For that reason the first hardware specialties were a fast 
multiply-add instruction, followed within a matter of months 
by hardwired correlators. In 1966 IBM announced the 2938 
array processor and made its first installation at the begin¬ 
ning of 1968. 

The 2938, designed in collaboration with Western Geo¬ 
physical, had an instruction set of about 12 instructions with 
data arrays as operands. Pipelining and other hardware in¬ 


novations resulted in great speed improvements. Geophys¬ 
ical processing could, as a consequence, proceed for almost 
a decade without serious hardware constraints by taking 
advantage of normal GP computer developments to comple¬ 
ment the array processor. 

Today’s array processors come in a bewildering variety 
of styles and capabilities. Married to a mini, one of today’s 
better array processors will, on seismic data, outperform a 
GP computer and do it at less than 25 percent of the cost of 
the GP machine. 

Seismic reflection data today, and for some time to come, 
consist of observations arrayed in a three-dimensional ma¬ 
trix. One dimension of the matrix, the one along the line of 
survey, is essentially unlimited but, as a practical matter, is 
limited to a few thousand values. The vertical (or time) 
dimension of the matrix is usually about 3000 data values; 
and the third dimension, representing the simultaneous ob¬ 
servations for each input pulse, is in the process of growing 
from a present typical value of 50 to about 500 or 1000 in 
the next few years. For some applications in which an area 
is surveyed in detail (so called 3-D surveys), a fourth di¬ 
mension of a few tens or hundreds of values is appended. 

It is becoming more common to use extended inputs such 
as coded or pseudo-random pulse sequences—a process 
which greatly increases raw data volumes. 

The only way all of these data will ever be processed is 
by extending the technology of specialized devices exem¬ 
plified by the array processor. 

I visualize the processor of 1985 built around a gigabyte 
high-speed memory with multiple access modes. Only a 
handful of algorithms will have to be applied to the data. 
The processor will have hundreds or perhaps thousands of 
identical single-chip microprocessors to execute each of the 
required algorithms. The micros will be used in a combina¬ 
tion of parallel and pipeline modes so that all are operating 
simultaneously. All of this activity will be under control of 
a relatively small CPU which can handle traffic control and 
the calculation of processing parameters. 

In operation all or a sizable chunk of the raw-data matrix 
will be read into main memory from the compact storage 
medium on which they were recorded in the field. A succes¬ 
sion of operators will be passed through the matrix in various 
directions and various orders by directing sequences of data 
from memory through the micros. 

Technology visible today can make the whole project fea¬ 
sible; and as Jules Verne has so aptly said, “What one man 
can imagine another man can do.” 
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INTRODUCTION 

The energy crisis has created increased interest in petroleum 
exploration. Discoveries of major deposits are increasingly 
rare and difficult to locate. The geophysicist requires more 
accurate knowledge to assist him in locating deeper and 
smaller hydrocarbon deposits. While gravity, magnetic, and 
electrical methods are also used, the predominant explora¬ 
tion tool is the seismic method and provides a wealth of 
subsurface information. Geophysicists continue to increase 
their seismic proficiency by using such things as modeling. 

The concern of this paper is the ability to model the 
seismic response of a known subsurface. The practicality 
and usefulness of a modeling technique is determined by 
how accurately, routinely, and economically it may be used 
to simulate realistic cases. Modeling techniques, such as 
numerical solutions to the wave equation and ray tracing, 
are possible and offer distinct advantages and disadvantages. 
The method presented in this paper is a numerical solution 
to the wave equation obtained by finite difference tech¬ 
niques. 

A brief background in seismic methods and theory is pre¬ 
sented. The numerical modeling technique is described, and 
a typical example is given. Also discussed is a minicomputer 
system which makes the method economically feasible for 
routine use. 

GEOPHYSICAL SEISMIC METHOD 

The typical method of gathering seismic data is depicted 
in Figure 1. At the site of interest, seismic wave transducers, 
or seismometers, are coupled to the surface of the earth 
along a line typically 10,000 feet in length. Individual seis¬ 
mometers (actually arrays) are generally spaced about 100 
feet apart and are monitored by a central recording station. 
A seismic source, often an explosive charge buried beneath 
the surface, generates seismic waves which travel through 
the subsurface according to the principles of elastic wave 
propagation. The direct wave and the many reflections from 
the subsurface interfaces are digitally recorded at the surface 
for several seconds. These records contain complex multiple 
reflections and ambient noise; they are later subjected to 
digital processing in an attempt to remove these effects. 

The records are often plotted as a seismogram as shown 


in Figure 2. Here the abscissa represents the horizontal 
position of the semismometers and the ordinate represents 
time. Time increases downward and is equal to zero at 
source initiation. The contribution to the plot from a single 
seismometer is highlighted. It is informative to notice the 
reason for this orientation of the display. It is a first attempt 
at inferring the geological structure from the recorded time 
response. A sense of horizontal position is somewhat pre¬ 
served while the time ordinate is related to the geological 
depth. 

Ideally, one would like to directly generate a picture of 
the subsurface geology from the recorded surface data. This 
would be referred to as the inverse problem. Unfortunately 
at this time it is only possible to deduce the structure, in¬ 
directly, after an involved sequence of digital processing, 
followed by extensive human interpretation. Mathematical 
modeling can be extremely helpful during the interpretation 
and processing steps. 

While the microscopic composition and porosity of the 
rocks is important, we are concerned only with their mac¬ 
roscopic effects upon seismic wave propagation. The earth’s 
subsurface, for our purposes, may be divided into homo¬ 
geneous regions characterized by constant velocity and den¬ 
sity values. This defines the subsurface in terms of the 
parameters in the wave equation. The variation of these 
parameters at the interfaces between regions gives rise to 
reflections, refractions, and other wave phenomena. 

FINITE DIFFERENCE METHOD 

Given a governing equation for seismic wave propagation, 
it is possible to numerically approximate its solution by finite 
difference techniques. Although the elastic wave equation 
governs seismic wave propagation, the simpler acoustic wave 
equation provides useful and practical results and will be 
considered here. The same approach has been applied to the 
elastic equation. 1 

The two-dimensional acoustic wave equation to be con¬ 
sidered is 



where v(x,z) is velocity, p(x,z ) is density, Uis the acoustic 
pressure, t is time, and x and z are horizontal and vertical 
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position, respectively. Interfaces are automatically ac¬ 
counted for since velocity and density variations are in¬ 
cluded in the equation. 

The majority of the calculations occur in regions with 
constant density, where equation (1) simplifies to 


r a 2 d 2 1 d 2 
[i? a+ 3F v ] ~ a? u - 


( 2 ) 


For simplicity, attention will be limited to equation (2). 
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Figure 2—Synthetic seismogram 


Using standard finite difference methods, 2 equation (2) 
may be approximated by 

U(x,z,t+At)= (1-2 p 2 )U(x,z,t) 

+ p 2 {U(x+ h,z,t)+ U(x-h,z,t) 

+ U(x,z+ h,t)+ U(x,z- h,t )} 


where 


- U(x,z,t—At), 


p(x,z)= 


v(x,z)At 
h ’ 


(3) 


where At is the time increment and h is the spatial grid 
interval. 

This finite difference scheme is one of many possible 
approximations. Notice that it provides an explicit solution 
for the acoustic pressure at a future time, t+ At, in terms of 
the pressure at the current time, t, the previous time, t-At, 
and the geological paramters. 

The geology to be modeled is represented by two-dimen¬ 
sional arrays of velocity and density as shown in Figure 3 
(h- Ax=Az). A uniform square grid is superimposed on the 
model, and at each discrete grid point, values for the velocity 
and density are assigned. The creation of these discrete 
arrays describing complex geological structures is well 
suited for interactive graphics programs. 

Consider the “molecule” of data required for the calcu¬ 
lation of the pressure field at a single point. The velocity 
and density parameters do not vary as a function of time, 
whereas the field value does. The algorithm may be visual¬ 
ized as a single matrix of geological parameters superim¬ 
posed on many multiple time planes of field values. Fortu¬ 
nately for storage conservation, the algorithm requires that 
only two successive time planes ever be present, since the 
future value can replace the previous value in storage. 

The accuracy of the algorithm is a function of the coarse¬ 
ness of the grid. For our purposes, a minimum of ten grid 
points per wavelength is usually sufficient. 3 There is also an 
upper limit imposed upon the time step to maintain the 
stability of the algorithm. 


T + 2 &T 



Figure 3—Geological parameters in discrete space superimposed upon 
multiple time planes of the wave field 
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ax = AZ = 25 FT. 

AT = .001 SECONDS FOR 2.5 SECONDS 

Figure 4 —Geological model 


MODELING APPLICATION 

Figure 4 depicts the two-dimensional geological cross sec¬ 
tion of a model. The small semicircular shaped region near 
the center represents a gas reservoir. A synthetic seismic 
response is desired in order to better understand how such 
a deposit would be observed in the presence of the other 
geology. 

The model extends 14,000 feet horizontally and 10,000 
feet vertically, which for a grid point spacing of 25 ft, results 
in 560 grid points horizontally and 400 grid points vertically. 
The response is generated for 2500 milliseconds to assure 
that reflections from the depth of interest are obtained. This 
requires about 2000 time steps. The time variation of the 
source is represented by the second derivative of a Gaus¬ 
sian, 3 with an upper-half-power frequency of 30 Hz, which 
is similar to that generated by an explosive charge in the 
field. 

Figure 2 is a plot of the synthetic time section obtained 
by recording the numerical values at the surface of the 
model. This corresponds to what the seismometers would 
record in the field. The first event to arrive at the surface is 
the direct wave from the source. Next is the reflection from 
the first interface, followed by many additional primary and 
multiple reflections. Although the geological model is rela¬ 
tively simple, the time section is complex and contains many 
refraction, reflection, and diffraction events. In practice, the 
responses for a large number of source locations are com¬ 
bined through an involved processing sequence, to create a 
more easily interpretable seismic section. An example of 
such a process, applied to a collection of synthetic seismo¬ 
grams for differing source locations, is shown in Figure 5. 
This synthetic section now more closely resembles the 
model geometry, and the lower boundary of the gas reservoir 
is readily identifiable as indicated on the display. Processing 
and interpretation of synthetic data sets simulating the actual 



Figure 5—Composite of multiple seismograms after processing 


field procedure provides an opportunity to develop and test 
ideas in a controlled, known, noise-free environment. 

An added bonus of this modeling method is “snapshots,” 
shown in Figure 6. A snapshot is a plot of the entire wave 
field at a single time plane. The geological structure has 
been superimposed on the display to assist the interpreta¬ 
tion. The snapshots are resampled by four to one to conserve 
storage, thus causing the structure to appear coarse. 

Snapshots can improve our understanding of seismic wave 
phenomena, since individual waves can be followed as they 
propagate through the media. A movie of seismic wave 
propagation has been produced using closely spaced snap¬ 
shot sequences. 

The finite differences technique is a very useful and flex¬ 
ible method for seismic modeling. Models of arbitrary geo¬ 
logical complexity require no unusual considerations and 
seismograms may be generated routinely. The synthetic re¬ 
sponse provides realistic waveforms with accurate arrival 
times and amplitudes. The response includes diffraction 
events and, if desired, the method can be applied to the 
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Figure 6—Snapshot sequence with structure superimposed 


elastic case to study shear wave events and other elastic 
phenomena. The flexibility of the method allows source and 
receiver locations to be easily varied, permitting complex 
arrays and non-standard acquisition techniques to be simu¬ 
lated. 


DEDICATED MINICOMPUTER SYSTEM 

The computational burden imposed by this modeling 
method is tremendous. Seismic models ranging in size from 
400 to 800 grid points horizontally and vertically are desir¬ 
able. This results in 160,000 to 640,000 grid points per model. 
Each geological point modeled requires from 10 to 20 bytes 
of computer storage. Typically the solution at 1500 to 3000 
time planes must be computed. Even though a maximum of 
two successive time planes are needed, this requires large 
quantities of storage. If sufficient local memory is not avail¬ 
able, computations may be performed on model segments 
rolled in and out from mass storage peripherals. 

Early finite difference modeling was performed on large 
general purpose computers using standard FORTRAN pro¬ 
grams. It was considered economically impractical to rou¬ 
tinely submit models larger than 100 by 100 grid points on 
such a system. Minicomputers were considered as an alter¬ 
native processing system. 

The dominant factor in the computational speed is the 
large number of wave field calculations required. Calcula¬ 
tions are performed in single precision floating point arith¬ 
metic. Figure 7 compares the floating point multiply and add 
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Figure 7—Multiple (*) and add (+) execution time for various floating point 
processors 


speed for memory to register instructions for a variety of 
processors. The processors may be divided into three cat¬ 
egories: large general purpose computers, minicomputers, 
and array processors. The array processors considered here 
are programmable devices that have their own local mem¬ 
ory. They require a host computer and, therefore, are con¬ 
sidered high speed computation extensions of the host. 

A dedicated minicomputer system, with minimal memory 
and peripheral devices, may achieve real time performance 
comparable to a large general purpose computer for appli¬ 
cations which are CPU bound, such as the one under con¬ 
sideration. In view of the comparatively modest hardware 
cost, the minicomputer system thus becomes economically 
attractive for installations where high and continued usage 
of the program is anticipated. The preceding modeling ex¬ 
ample required approximately eight hours of minicomputer 
CPU time (without an array processor) to create the syn¬ 
thetic seismogram for a single shot location. This would 
have been impractical on a large computer. 

An Interdata 8/32 minicomputer was selected: it can ser¬ 
vice any subordinate function required and is capable of 
hosting an array processor, making available their powerful 
resources. The hardware configuration used is shown in 
Figure 8. The minicomputer has 32 bit architecture and 
hardware implemented floating point instructions. It sup¬ 
ports up to one megabyte of memory with a bandwidth 
greater than six megabytes. Multiple selector channels allow 
the peripherals high speed DMA (Direct-Memory-Access) 
capabilities. This is important since high speed mass storage 
discs are an integral part of our implementation. The op¬ 
tional array processor also utilizes DMA extensively. A 
multiplexor bus is available for other low-speed peripherals. 

The speed of the main calculation loop is paramount to 
overall performance. While the bulk of the program may be 
written in FORTRAN, this loop must be carefully optimized 
in assembly language. This is especially important with some 
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minicomputer FORTRAN compilers, which may not pro¬ 
duce very efficient execution code. This relatively simple 
improvement has yielded a 4-fold speed improvement. The 
same idea applies if an array processor is used; it is more 
efficient to program the algorithm directly on the array pro¬ 
cessor, than to use standard vector multiply and add se¬ 
quences. Not only may direct programming improve speed, 
but it can also conserve expensive array processor local 
memory. 

The continuous flow of model segments from storage, to 
memory, and back to storage again, is an integral part of the 
processing. One must be acutely aware of operating system 
overhead and I/O requirements for optimum performance. 
The hardware configuration has two high speed mass storage 
discs, each with an independent DMA selector channel. An 
unbuffered disc file access method provides the most effi¬ 
cient I/O. This allows input, output, and calculation of model 
segments to proceed concurrently. This is considerably 
faster and more efficient in real time than if each occurred 
serially. The ability to do simultaneous I/O and calculations 
emphasizes how the user may closely coordinate software 


and hardware. By carefully controlling the disc random ac¬ 
cesses, the calculations may be synchronized to the point of 
transferring model segments on successive fractional rota¬ 
tions of a high speed disc. 

CONCLUSIONS 

Seismic modeling using finite difference methods provides 
the exploration geophysicist accurate synthetic results for 
complicated geological structures. Modeling provides a 
means of comparing synthetic results with field data. It 
provides a fast, flexible, and economical laboratory for in¬ 
vestigating new procedures and confirming old ideas. The 
finite difference approach used here is but one of many 
numerical variations possible, all of which merit further con¬ 
sideration. This technique is not limited to geophysical ap¬ 
plications, but is applicable to other areas of wave propa¬ 
gation research. 

Programs demanding heavy CPU computations for long 
periods of time may not be justifiable on large general pur¬ 
pose computers. Large time shared computers are designed 
to obtain high usage levels through simultaneously and ef¬ 
ficiently managing multiple tasks. Demanding programs, 
such as finite difference modeling, utilize the CPU for long 
periods of time which eliminates the need for sophisticated 
multiple user management. Dedicated minicomputer sys¬ 
tems may be more economical because the memory and 
peripheral devices can be tailored to the specific application. 
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Present and future applications of computer technology to 
petroleum exploration 

by R. A. TERNUS and A. S. HOFFMAN 

Chevron Oil Field Research Program 
La Habra, California 


INTRODUCTION 

In this paper we will summarize the steps involved in ex¬ 
ploring for oil and gas, review the role that computers play 
today in exploration and discuss some of the roles that 
computers may play in the future. 


PETROLEUM EXPLORATION 

The ultimate way in which oil and gas are discovered is 
to drill into the ground. The objective of all prior steps in 
the exploration process 1 is to identify where to drill to min¬ 
imize the risk of drilling “dry” holes. 

Generally exploration (Figure 1) begins with a regional 
geologic survey. This is done to locate geologic basins con¬ 
taining sedimentary rocks, and to extract information from 
surface geologic features and rock samples about the types 
of sedimentary rocks and their depositional environments. 
Such information can suggest where oil and gas may have 
been generated and is trapped in reservoirs. 

The second step may consist of gravity and magnetic 


surveys. These determine the regional variations in the 
thickness of the sedimentary rocks and locate subsurface 
structures such as faults and salt domes, which often help 
to form traps for hydrocarbons. 

Gravity prospecting involves measuring local fluctuations 
in the earth’s gravity field, then correcting these for latitude, 
elevation, topography, and earth tides. The resulting gravity 
maps can be useful in determining subsurface geological 
features marked by changing rock density. 

Magnetic prospecting, which measures local fluctuations 
in the earth’s field, is more complex and erratic than gravity 
measurements. However, the magnetic measurements are 
simpler and cheaper to make, require only minimal correc¬ 
tions, and are more diagnostic of mineral distribution. 

The third step consists of reflection seismic surveys, some 
of which may be long regional lines used to better define the 
sedimentary basin. Others are laid out on more finely-spaced 
grids—to obtain detailed information in specific areas. From 
these one can determine the specific locations of possible 
hydrocarbon traps and, in some cases, the actual presence 
of hydrocarbons. 

The basic principles (Figure 2) of the seismic reflection 
method 2 are rather straightforward. A controlled source of 



Figure 1—The exploration process 
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Figure 2—Illustration of the seismic reflection method 
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Figure 3—Gravity profile and geologic section 
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(a) ACTUAL GEOLOGIC STRUCTURE 



(b) DENSITY MODEL OF 
STRUCTURE 



(c) CORRESPONDING GRAVITY PROFILE 

Figure 5-A salt dome and its interpretation model 
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acoustic energy at or near the earth’s surface propagates 
energy into the subsurface. The reflected energy from geo¬ 
logic strata is sensed by arrays of seismic detectors at the 
earth’s surface (or in the sea, slightly below the surface). 

Correlating the resulting subsurface reflectivity maps with 
a knowledge of geologic structure and rock properties allows 
the interpreter to delineate a three-dimensional geologic 
model of the subsurface. The basic acquisition objective of 
the seismic technique is to maximize the contribution from 
compressional waves reflected from the depths of interest 
and to minimize that from surface waves and ambient noise. 

CURRENT USE OF COMPUTERS 

Computers are used extensively today in all phases of 
exploration, with the possible exception of the initial re¬ 
gional geologic surveys. They are used to filter the gravity 
and magnetic data, make various geometric-type correc¬ 
tions, and assist in interpreting the data (Figures 3 and 4). 
The latter is often accomplished by constructing mathemat¬ 
ical models that describe the shapes, densities, and/or mag¬ 
netic susceptibilities of various parts of the subsurface. The 
implied gravity and magnetic fields for these models are 
calculated and compared to measured fields. The mathe¬ 
matical models are then adjusted to obtain suitable matches 
between implied and actual fields. The final model (Figure 
5) provides an interpretation of the data. 


However, the most widespread use of computers in ex¬ 
ploration is in the seismic reflection method. There is a 
ubiquitous presence of various types of data-processing sys¬ 
tems, including data acquisition in the field, “number¬ 
crunching” data processing and reduction, and the often- 
interactive interpretive phase (Figure 6). 

The data acquisition stage (Figure 7) involves several 
thousand individual detectors connected in 100 or so sepa¬ 
rate recording channels. Each channel is sampled at up to 
1000 times per second, usually over the 4-6 seconds of useful 
reflection time for a single seismic source event. Depending 
on whether the acoustic source is below or on the surface, 
from about a hundred to several thousand such events can 
be initiated each day and either recorded or partly 
“stacked” in the field and then recorded. 

The objective in the processing stage 3,4 is to enhance the 
desired signal at the expense of the noise, to make geometric 
corrections to the data as well as to extract desired param¬ 
eter values. A typical processing flow (Figure 8) might be as 
follows: magnetic tapes containing the field data are played 
back in the processing center, and the data is sorted and 
properly formatted for processing. Traces, having either ex¬ 
cessive noise or neither noise nor data (i.e., dead traces) are 
edited out or flagged so that they do not interfere with future 
processing. 

Next, the data may be filtered to remove noisy parts of 
the frequency spectrum that contain little or no signal. Static 
adjustments (time shifts) are calculated and applied to each 
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COMMON DEPTH POINT (CDP) STACKING 
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NOISE AND OTHER SEISMIC EVENTS ADD OUT OF PHASE. 

Figure 10—Common depth point stacking 


trace, to compensate for variations in shot and receiver 
elevation and in the thicknesses of shallow, low-velocity 
sediments. The velocities in the subsurface are estimated by 
examining the different arrival times of reflections associ¬ 
ated with different shot-receiver offset distances (Figuare 9) 
but the same reflection point in the subsurface. Deconvo¬ 
lution—a filtering type of operation that serves to “whiten” 
the remaining part of the spectrum—is normally applied. 
This improves the resolution between closely spaced events. 
Finally, all traces having the same common depth point 
(Figure 10) may be summed to create a final stacked section. 
At various points in this processing stream, special tech¬ 
niques may be employed to enhance the coherent signals 
and to reduce multiple reflections. 

Computers are also used to interpret seismic data in a 
manner analogous to that described previously for gravity 
and magnetics data. 

As the data is collected and interpreted, maps and seismic 
cross sections are constructed (Figures 11 and 12) showing 


the configurations of the sedimentary rock layers (beds) and 
possible locations for trapped hydrocarbons. Interactive 
graphics terminals are often used so the interpreter can work 
with the computer in the construction of these maps. 

After these initial steps a well (or wells) may be drilled. 
Whether or not hydrocarbons are found in the drilling, data 
about the rocks and their pore fluids are incorporated with 
the previously obtained data, to refine the interpretations 
and maps and identify additional drilling locations. 


FUTURE USE OF COMPUTERS 

In the future the use of computers will continue to expand 
in the roles of processing and interpretation. Processing 
algorithms often are built around simplifying approxima¬ 
tions, made to hold down computer processing time. As 
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SEDIMENTARY STRUCTURES WHICH PRODUCE OIL TRAPS, (a) VERTICAL SECTION 
THROUGH ANTICLINE ALONG LINE MN IN (b); (b) MAP OF HORIZON A IN (a); 

(c) VERTICAL SECTION THROUGH FAULT TRAP ALONG LINE XY IN (d); (d) MAP ON 
TOP OF BED B IN (c); (e) PINCHOUT; (f) UNCONFORMITY TRAP; 

(g) OIL TRAPS ABOVE AND AROUND A SALT DOME, a) CAPROCK FIELD OF ANTICLINAL 
TYPE WITHIN THE TOP OF THE SALT DOME, b) FIELD OF ANTICLINAL TYPE IN LAYERS 
ABOVE THE DOME, c) FAULT TYPE FIELD AT THE FLANK OF THE SALT DOME, d) FAULT 
TYPE FIELD BELOW THE OVERHANGING TOP. e) FAULT TYPE FIELD BETWEEN TWO 
RADIAL FAULTS, f) FIELD BELOW A DISCONFORMITY, WHICH HAS BEEN CAUSED BY 
MOVEMENTS OF THE SALT, g) OVERLAP FIELD ABOVE SUCH A DISCONFORMITY. 
h) SHALING OUT FIELDS ON ONE SIDE OF THE SALT DOME. 

Figure 11—Typical exploration prospects 
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computers become more powerful, some of the approxi¬ 
mations can be eliminated, so we can improve our process¬ 
ing capabilities and thereby extract useful information from 
data which now yield little or none. In addition, we will be 
more finely sampling the seismic data. This will improve 
resolution, accuracy, and sensitivity in our interpretations 
of the subsurface by increasing the bandwidth of the data. 

While much of this processing is done on large general- 
purpose computers, we have, over the last few years, begun 
to make more use of special-purpose computers, such as 
programmable array processors, and this trend is certain to 
continue. The use of interactive graphics is still in its in¬ 
fancy, and we will see considerable growth in the use of this 
type of computer aid. Consider, for example, the problem 
of reordering seismic data. Some of the processes mentioned 
above operate on one trace at a time. Others need all of the 
traces originating from a single shot. Still others need all of 
the traces having a common depth point; that is, those hav¬ 
ing the same midpoint between their respective shots and 
receivers. As we work with more traces, because of the 
need for higher redundancy, we will need a “black box” 
that can rapidly reorder the data prior to the individual 
processing steps. 

Preprocessing during data collection will also require more 
special-purpose systems. Consider the current limitations of 
some of our data collection procedures. In a normal seismic 
survey, multiple (10-100) sensors, i.e., geophones or hydro¬ 
phones, are connected to collect the data for a single trace. 
This increases the signal-to-noise ratio at the output of the 
sensor group over that available from the individual sensors 


and reduces (by spatial filtering) the effect of unwanted 
energy propagating along the surface. However, this type of 
configuration has limitations, particularly in recording higher 
frequency seismic data. In the last few years, we have been 
going in the direction of recording more groups, so that the 
length of each can be reduced. 

Ultimately we may record each sensor individually. How¬ 
ever, this leads to the collection of large volumes of data 
that must be transported back to the processing center and 
put through on the computer. This method may be limited 
to areas requiring the highest data quality obtainable. 

In the next five years one can project data acquisition 
rates up to 40 MBits/sec, requiring a storage capacity per 
record (i.e., a single seismic event for a deep source, or 
twenty or so coherent events for a surface source) of 350- 
500 MBits. For normal crew operations, this would require 
a storage capacity of from 10-350 GBits/day’s operation. 
Obviously we will need either a mass storage mechanism in 
the field, or more processing before recording, or possibly 
both. 

In order to obtain many of the benefits of recording each 
sensor individually, we will probably go in the direction of 
preprocessing at the group level in the field (Figure 13) 
rather than in the processing center. The rapid growth of the 
computer hardware technology is well-known, particularly 
the increasing capabilities of micro-processors and the in¬ 
creased density of memory chips. It is reasonable to expect 
that within the next few years we will develop the hardware 
and software algorithms necessary to perform differential 
dynamic corrections, differential static corrections, and 
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Figure 13—Group level preprocessing 


waveform variation adjustments on the output signals from 
the individual sensors. We can then sum these signals in the 
field to obtain the equivalent of today’s sensor group output. 

In summary, computers are used extensively to assist in 
the exploration for oil and gas. Future processing and inter¬ 
pretation requirements will put greater demands on our use 
of computer technology, and we will have to use innovative 
approaches in order to solve these problems. 
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Remote sensing and image processing in earth resources 


THE EARTH RESOURCES SITUATION 

Population growth and demands for higher standards of living are creating 
pressures which affect all aspects of our society. The global population, currently 
near 3900 million, is growing at a rate of 75-80 million/year. 

Coupled with population growth, and magnifying the effect, are rising demands 
for a more equitable sharing of the products and benefits of technology. The 
products and benefits derive from the consumption of natural resources. As 
societies develop and technological advancement begins, individual demands 
increase resulting in a more rapid rate of consumption of resources. Information 
is necessary to enable prudent decisions regarding the management of resources. 
But understanding alone will not lead to solutions on a long-term basis. Knowl¬ 
edge must be current, necessitating continuing inputs of fresh information. The 
most practical and economical manner of acquiring repetitive global, Earth Re¬ 
sources information is by means of a satellite-based remote sensing system. 

Key to the utilization of the data is the ability to register the data to the ground, 
to previous images, and to data derived from other sources, and then to perform 
analysis. These together comprise the area of image processing. 


IMAGE PROCESSING 

The area as a whole will be covered by four sessions. They will be planned to 
lead persons not knowledgeable in image processing into the subject, with primary 
emphasis on image processing for remote sensing of the Earth. This area of 
application is selected because of its growing importance in the maintenance of 
our quality of life and the growing need for monitoring of the Earth’s resources. 
The four sessions taken together will have some aspects of a short-short course, 
in that they will briefly introduce the remote sensing problem and the character¬ 
istics of images (Session I), discuss image-specialized computer techniques and 
programming considerations (Session II), then considerations of image processing 
system design and digital image based information systems (Session III). Session 
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IV is planned to be a panel, with presentations of future satellite and data plans 
and data characteristics. This is planned as a panel to allow rebuttals and dis¬ 
cussions of these future plans and their implications to the computer world. 

This format has been selected to provide cohesive treatment of the subject, 
with the thought that many computer people who have not been involved in it 
will become involved in the future as the popularity of digital image processing 
and analysis grows. This growth is expected based on the widespread use of the 
digital data to date, far beyond the early NASA expectations, and the NASA 
plans to provide much improved digital data in the future. 

Applications have been discussed in other contexts extensively. The discus¬ 
sions here will lay the groundwork, but then concentrate on the implications to 
the computer world. These implications will include data timeliness, effects of 
various processes such as the necessary geometric registration on the utility of 
the data as seen during the subsequent analyses, and the user needs for interaction 
and data display. 

In the applications world, images are getting larger, for example, up to 
6000x 6000 multiband elements for future Landsat. Data storage, dissemination, 
and processing are all going to be a problem, especially to the small or medium 
user. At the same time, the analyses being attempted are becoming more sophis¬ 
ticated. There is a need for image descriptive and image processing languages 
which will provide improved tools for the analyst. They must be flexible, efficient, 
and easy to use. New data dependent adaptive techniques are being devised. 

The required image analysis cannot exist in a vacuum. Efficient systems must 
be provided. Three types are needed: (1) Large pipe-line systems for handling 
large quantities of data routinely and quickly; (2) Flexible, modular systems for 
research and/or interactive analysis; (3) Geographically oriented systems for use 
with/as geocoded data bases. These all have features in common: need for effi¬ 
ciency, need to retrieve and handle large multiple data sets (the images), possible 
need for interactive analysis and refreshed image displays. The latter two also 
have the need to be easily modularly expanded and updated. The geocoded 
systems must be able to input and output data in tabular and polygon form as 
well as imagery, and be able to use all together. 




Information requirements for natural resource inventories 


by WILLIAM J. BONNER 

Bureau of Land Management 
Denver, Colorado 


INTRODUCTION 

Background 

The Bureau of Land Management (BLM) administers the 
natural resources on 191 million hectares of public lands in 
eleven* contiguous western states and Alaska. In addition 
BLM has management responsibility for 25 million hectares 
of publicly owned mineral resources on privately owned 
lands. Added to this is the management responsibility on 
approximately 500 million hectares of Outer Counter Shelf 
(OCS) lands. 

The basic task of BLM is to manage all resources found 
on these lands to provide maximum public benefit and full 
consideration for good conservation practices. This includes 
the dedication to carrying out whatever programs are re¬ 
quired to insure that the stewardship of the public lands and 
their resources leads to the optimum planned use for the 
long range public good. These programs include activities in 
domestic livestock grazing, fish and wildlife ecology and 
habitat development, outdoor recreation, timber production, 
watershed protection, wilderness preservation, minerals de¬ 
velopment, environment protection, river basin planning, 
and general land use classification under the concept of 
multiple use. Resource management and development activ¬ 
ities are supported by various programs which provide for 
roads, trails, and physical improvements such as recrea¬ 
tional facilities; and by an active program to protect the 
public lands and their resources from wild fires and from all 
forms of public and private misuse. 

To carry out these programs BLM must collect and handle 
large quantities of resource inventory information. Further 
the fiscal and manpower resources which must be committed 
to gathering this information is staggering. This is due to (1) 
the increasing public awareness of the importance of wild¬ 
lands as a national resource, and (2) the increasing need for 
documentation of the bases of management actions, includ¬ 
ing baseline resource conditions, trends, plans, and environ¬ 
mental analysis of proposed actions. For example, a large 
fraction of the vital energy resources (coal, oil, oil shale) of 
the nation underlie the public lands; the development of 


these resources in a manner consistent with national inter¬ 
ests requires timely, accurate, and comprehensive resource 
inventory information. 

In October 1976, the 94th Congress passed Public Law 
94-579. The purpose of this Act is to establish public land 
policy; to establish guidelines for its administration; and to 
provide for the management protection, and enhancement 
of the public lands. In addition this Act which is known as 
the “Federal Land Policy and Management Act of 1976” 
charges the Secretary of the Interior with the following re¬ 
sponsibility.* 

“The Secretary shall prepare and maintain on a continuing 
basis an inventory of all public lands and their resource 
and other values (including but not limited to, outdoor 
recreation and scenic values), giving priority to areas of 
critical environmental concern. This inventory shall be 
kept current so as to reflect changes in conditions and to 
identify new and emerging resource and other values. The 
preparation and maintenance of such inventory or the 
identification of such areas shall not of itself, change or 
prevent change of the management or use of public 
lands.” 

Thus Public Law 94-579 coupled with existing requirements 
causes BLM to reassess its traditional methodologies for the 
collection of resource inventory information. 

Remote sensing in resource management 

BLM is involved in a program with the National Aero¬ 
nautics and Space Administration (NASA) and EROS Data 
Center (EDC) to evaluate remote sensing technology. The 
purpose of this evaluation is to determine the feasibility of 
performing resource inventory via remote sensing on an 
operational basis. 

Management of the public lands under the concept of 
multiple use requires consideration of several resources. 
The BLM system identifies these resources as: range, for¬ 
estry, wildlife, recreation, watershed, minerals, and lands. 
The common denominator of these resources is vegetation 


* Arizona, California, Colorado, Idaho, Montana, Nevada, New Mexico, 
Oregon, Utah, Washington and Wyoming 
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and for that reason BLM has selected the broad area of 
Wildland Vegetation as the theme for this program. The 
overall program objective is to test and implement a remote 
sensing based inventory system for wildland vegetation re¬ 
sources on an operational basis within BLM. It is planned 
to operationally implement the techniques developed during 
this project. 


Agency roles 

The Wildland Vegetation Resource Inventory Project is 
a joint effort between the three previously mentioned organ¬ 
izations. The BLM has overall management authority in 
the program and works closely with NASA and EROS to 
insure that project schedules and milestones are met and 
that program activities are successfully accomplished. In 
addition to this management function the BLM assumes the 
following responsibilities: 

(1) Coordinate and document user activities within BLM. 

(2) Perform cost analysis studies. 

(3) Acquire large scale photography. 

(4) Specify and evaluate output products. 

(5) Implement training programs in remote sensing for 
BLM personnel. 

NASA, working in cooperation with BLM, is providing 
the technical capability for the initial year of the program 
through a contract with a commercial remote sensing firm. 
As part of this contract NASA is obtaining data analysis, 
documentation, and specialized training for BLM personnel. 
In addition NASA is providing small and medium scale 
photography to be used as supporting data in the digital 
analysis processes. 

EROS through EDC provides specialized facilities and 
equipment as well as technical support for key project ac¬ 
tivities. In addition EDC is collaborating with BLM on de¬ 
velopment and implementation of a BLM wide remote sens¬ 
ing training program. 

The overall concept of NASA and EROS involvement is 
to transfer remote sensing technology to BLM. To achieve 
this goal a plan has been developed in which BLM rapidly 
assimilates the technology in house and assumes prime re¬ 
sponsibility for performance of the project tasks. At the end 
of the program BLM will have an operational program in 
remote sensing and be independently capable of performing 
all activities required for wildland vegetation resource in¬ 
ventory. 


TEST SITES 

In selecting test sites for this program a number of factors 
were considered. It was desired to have test sites with a 
variety of vegetation. In addition test sites were to be rep¬ 
resentative of the public lands which BLM administers. The 


size of the test site was scoped at approximately one million 
hectares; a size judged sufficient to constitute a legitimate 
test of the technology but not so large as to require more 
than one project year to complete. The final criterion for test 
site selection was the presence of timber in sufficient volume 
to conduct a meaningful test of multistage sampling tech¬ 
nology. Using these guidelines the BLM selected test sites 
in Alaska, Arizona and Idaho. 


Alaska test site 

The Alaska test site is located in the interior of Alaska 
and encompasses approximately one million hectares. The 
northern boundary of the site is roughly the foothills of the 
Alaska range. The southern boundary is defined by an east- 
west line passing through the confluence of the Maclaren 
and Susitna Rivers. The test site extends from the town of 
Paxson on the east to Cantwell on the west. The Denali 
Highway which is 135 miles in length transects the site. The 
range of vegetation in the test site includes all vegetation 
associations normally found in interior Alaska. Typical veg¬ 
etation units are black spruce, alder, willow, and dwarf 
birch. Tundra units include many varieties of lichens, moss 
and grass. The test site is in an area which is known as the 
“Denali,” the Athabascan Indian name for Mt. McKinley, 
meaning “The Big One.” The Alaskan Test Site is typical 
of the northern spruce-tundra biome. 


Arizona test site 

The Arizona test site is located in the extreme northwest 
comer of the state of Arizona in the BLM Arizona Strip 
District. The test site is approximately one million hectares 
in size. It is bordered on the west and north by the states of 
Nevada and Utah respectively. Its eastern boundary is range 
line 6W and 7W and the Colorado River forms its southern 
terminus. The test site is approximately 88 km southeast of 
the town of St. George, Utah. The test site characterizes the 
southwest desert community and includes vegetation types 
common to that unit; such as, creosote, big sage, and black¬ 
brush. In addition significant stands of pinyon-juniper and 
ponderosa pine are found in the higher elevations. 


Idaho test site ' 

The Idaho test site is in the southwest comer of the state 
of Idaho and is also comprised of approximately one million 
hectares. It is bordered on the north by the Snake River, on 
the east by the Boise Meridian, on the west by the state of 
Oregon and on the south by the state of Nevada. The test 
site typifies the extensive sagebrush-grassland communities 
of the inter mountain basins. Vegetation in the test site is 
primarily grass and sage with some douglas fir and pinyon- 
juniper stands in the higher elevations. 
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RESOURCE INVENTORY REQUIREMENTS 
Landsat data processing 

BLM is utilizing an interactive digital image analysis sys¬ 
tem for processing of Landsat data. This system is capable 
of processing digitized multispectral scanner (MSS) data and 
has a “stand alone” image processing capability which pro¬ 
vides a wide range of digital image processing techniques. 
The system is configured to utilize a minicomputer and is 
user interactive, with an interactive color display. The sys¬ 
tem uses a maximum likelihood algorithm for Landsat clas¬ 
sification. The hardware and software configurations are 
readily expandable for maintaining the system at the state- 
of-the-art and for meeting unique data requirements. 

Preprocessing 

BLM applies the following radiometric corrections to the 
Landsat data. These corrections are applied as appropriate 
depending upon the task to be accomplished. These radio- 
metric corrections are: line drop, banding removal, noise 
filtering, atmospheric path radiance, and sun angle. 

In addition to the previously mentioned radiometric cor¬ 
rections, the following geometric corrections are applied: 
mirror scan velocity profile, detector sampling delay, pan¬ 
oramic distortion, scan skew, earth rotation, spacecraft ve¬ 
locity, perspective geometry, attitude, altitude, map projec¬ 
tion, and adjustment to ground control. These corrections 
are applied to obtain an accuracy consistent with the base 
map. 

Classification for vegetation 

A maximum likelihood algorithm is preferred for the clas¬ 
sification process. BLM uses unsupervised, supervised and 
controlled clustering techniques for classification. Generally 
speaking controlled clustering is the approach most com¬ 
monly used in BLM, due to the spectral variability which is 
found in wildland environments. Once the digital data has 
been classified (in this case a final product is referred to) it 
is desirable to apply a “smoothing” or “cleaning” algorithm 
to reduce the “salt and pepper” effect of pixel by pixel 
classification. In addition BLM has a requirement to “gen¬ 
eralize” the classified data to provide representation of 
cover types for each map scale required. The classified data 
must also be quantified in terms of acreages for combina¬ 
tions of vegetative communities and their characteristics. 
These acreages must be compiled and tabulated for each 
class by total area and/or jurisdictional unit. 

Data must be environmentally stratified for such param¬ 
eters as topography, elevation, precipitation, and ecotypes. 
In addition administrative boundaries must be overlain on 
the Landsat data to the accuracy of the base maps used in 
the process. When appropriate BLM performs multidate 
processing to obtain additional information and accuracy 
from the classification process. Change detection is utilized 


to map physical changes resulting from wildfire, mining op¬ 
erations, and the like. It is also necessary to digitally mosaic 
adjacent Landsat scenes in order to produce a composite 
image of a study area. 

Accuracy determination 

In the past the evaluation of the viability of in place 
vegetation type maps produced through discriminant anal¬ 
ysis of digital data image processing technicians have been 
inhibited by a lack of “a posteriori” verification. Verifica¬ 
tion provides an index with respect to the utility of the 
product. This verification must be a statistically valid eval¬ 
uation of the accuracy of the classification and should be 
accomplished through a systematic or random selection of 
pixels from each class on the Landsat product. Pixels can 
be compared to classifications based on photo interpreted 
plots and ground samples to determine percent misclassified. 
Areas to be examined in the verification process must be 
chosen independently of the areas used in the classification 
process. The sampling design for verification must account 
for spatial errors due to pixel misregistration and the realities 
of collecting photography and ground data for the verifica¬ 
tion. The following items are acquired by BLM in verifying 
their classification products: 

(1) A percentage accuracy of classification for each cover 
type class, with a corresponding statistical confidence 
statement for each percentage. 

(2) A numerical summary for each cover type indicating 
the kinds of classification errors (omission and com¬ 
mission). 

(3) A narrative indicating probable causes of misclassifi- 
cation; e.g., spectral similarity, spatial resolution, and 
phenological phenomena, etc. 

(4) A definition of class names for cover types encoun¬ 
tered in the classification process through species 
composition determination. 

(5) A statistical confidence statement for each area by 
cover class determination. 


Enhancement for geology 

BLM uses both standard and enhanced Landsat products 
for the purpose of geologic interpretation. Enhanced prod¬ 
ucts are produced using a variety of digital image processing 
techniques; such as ratioing, contrast stretching, truncation, 
fourier filtering, etc. Both environmental and geomorpho- 
logic data are required. Examples of the parameters of sig¬ 
nificance from our program in Alaska are as follows: 

A. Environmental—structural geology 

1. Active volcanoes 

2. Geothermal hot spring areas 

3. Active glaciers, with aerial extent and whether ad¬ 
vancing or retreating 

4. Active landslide areas and potential hazard areas 
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5. Large active snowslide areas 

6. Severe erosion areas—classify genetically 

7. Lineaments interpreted to be of geologic, tectonic 
significance 

8. Geologic structures 

B. Geomorphologic 

1. Morainal features, classified as lateral or terminal 

2. Outwash plains, classified as well-drained or 
poorly drained 

3. Eskers 

4. Alluvium, classified as silt and clay, gravel, or 
unsorted 

5. Beach deposits 

6. Igneous features, which produce a significant 
landform such as dikes, intrusive igneous struc¬ 
tures with a topographic expression, volcanic 
cones, calderas, lava flows, etc. 

7. Karst topography 

8. Permafrost and nonpermafrost areas 

9. Active strip mining areas 

10. Bedrock geology—identify and map those remote 
sensing units which coincide with conventional 
geologic mapping units, i.e., formations, mem¬ 
bers. 


Multistage sampling 

Statistical sampling schemes, using multistage variable 
probability sampling for areal estimate of timber volume and 
rangeland productivity, are being used in this program in 
Arizona and Idaho. A Landsat digital classification is used 
as the stratification and small, medium, and large scale pho¬ 
tography are used as necessary with supplemental ground 
work. The BLM program attempts to address rangelands, 
forest lands, and woodlands. 

Rangelands 

For each Landsat classified rangeland vegetation type the 
following parameters are estimated: ground cover, species 
composition, vegetation condition, forage production, and 
grazing carrying capacity. Statistics are aggregated by pas¬ 
tures and allotment. 

Forest lands and woodlands 

For each forest stand identified in the stratification proc¬ 
ess the following parameters are estimated: size and age 
class, areal extent, volume, mortality. Statistics are aggre¬ 
gated for each stand.* These same parameters are estimated 


* A stand is an aggregation of trees or other growth occupying a specific area 
and sufficiently uniform in species composition, age, arrangement, and con¬ 
dition as to be distinguishable from the forest or other growth on adjacent 
areas. 


Level I 

Level II 

Level III 

Level IV 

Forest 

Conifer 

Dense Conifer 

Open Conifer 

White Spruce 

Black Spruce 
Black/White Spruce 
White Spruce 

Black Spruce 
Black/White Spruce 


Deciduous 

Dense Deciduous 

Sparse Deciduous 

Aspen 

Birch 

Aspen/Birch 

Cottonwood 

Aspen 

Birch 

Aspen/Birch 

Cottonwood 


Mixed Forest 

Deciduous Mix 

Coniferous Mix 

Birch/Spruce 

Aspen/Spruce 

Cottonwood/Spruce 

Spruce/Birch 

Spruce/Aspen 

Spruce/Cottonwood 


Figure 1—Example of provisional classification framework for Alaskan 
vegetation 


for woodlands but since the concept of a stand for wood¬ 
lands is not well defined, aggregation is by allotment. 


Classification Framework and Platforms 

For this project BLM has developed classification frame¬ 
works for use with remote sensor data. The classification 
system developed is suitable for use with data from both 
satellite and aircraft. Figures 1 and 2 are examples of the 
Provisional Classification framework for Alaska and Arizona 
respectively. Using Anderson (1976) the multilevel classifi¬ 
cation system indicates the fact that different sensors pro¬ 
vide data at a range of resolutions dependent upon altitude 
and scale. In general, the following relations are reasonable: 
Level I—Landsat, Level II—High altitude aircraft, Level 
III—Medium altitude aircraft, Level IV—Low altitude air¬ 
craft. It is the goal of BLM to develop guidelines, based 
upon the foregoing, which indicate the scales and levels of 
data suitable for given management requirements. BLM is 
also attempting to push the different levels of data, such as 
Landsat, to its maximum extent hoping to obtain good eval¬ 
uations of the satellite as a mixed level II, III, and IV sensor. 
Figure 3 illustrates vegetation classification level versus 
scale and source of data. 
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Pinyon 
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Salt Cedar 
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Figure 2—Example of provisional classification framework for Arizona vegetatioi 



Information Requirements for Natural Resource Inventories 


91 


Class 

Example 

Scale 

Source 

I 

Tundra 

1:250,000 

Landsat 

II 

Dry Tundra 

1:120.000 

High Altitude Photography 

III 

Closed Mat and 
Cushion 

1:30,000 

Medium Altitude Photography 

IV 

Dryas Meadow 

1:6,000 

Low Altitude Photography 


Figure 3—Classification level versus data source 


OUTPUT PRODUCT REQUIREMENTS 

A requirement exists for a variety of output products. 
Color coded classified maps adjusted to a Universal Trans¬ 
verse Mercator (UTM) projection are required. These maps 
are to a scale of 1:250,000 and are aggregated in eight or 32 
hectare units as required. Polygon maps showing vegetation 
as sets of closed polygons are also used. Polygon maps are 
typically aggregated in eight hectare units. Another product 
which BLM has found to be beneficial is the line printer 
map which is typically at a scale of 1:24,000. In this product 
the aggregations are dependent upon print character size. 
Other products which are obtained from Landsat data are: 
geologic maps showing environmental, structural and geo- 
morphological data; surface water maps showing clear and 
sediment laden water; fire hazard maps showing vegetation 
types and spread rates. 

SUMMARY 

BLM is responsible for the multiple use management of the 
public lands. Public Law requires that a continuing and 


current inventory be maintained on the public lands and 
their resource values. In order to meet the requirements of 
law, as well as to insure that the stewardship of the public 
lands and their resources leads to the optimum planned use 
consistent with the long range public good, BLM is evalu¬ 
ating remote sensing as an operational tool for resource 
inventory. 

The remote sensing program and its attendant activities de¬ 
scribed in this paper attempt to define methodologies to 
acquire quality resource inventory data. Vegetation units 
are mapped, acreages are compiled. Multistage sampling 
techniques are used to estimate timber volume and standing 
forage. Change detection methodology is used to map phys¬ 
ical change from wildfire and mining activities. Geologic 
maps are produced from sophisticated image processing 
techniques. All of the resultant data are verified using sta¬ 
tistically valid processes. Products are evaluated and inter¬ 
preted by qualified resource specialists who have also re¬ 
ceived intensive training in remote sensing. The result is the 
development within BLM of information requirement guide¬ 
lines defining platform levels (satellite; high, medium, or low 
altitude aircraft) suitable for selected resource inventory 
tasks. 
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for natural resource inventories* 
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INTRODUCTION 

Landsats-1 and -2 have provided Earth resource scientists 
the opportunity to acquire multispectral data repetitively 
over large regions. These data are being used to monitor 
change in Earth resources, map resources over extensive 
regions, and inventory specific resource parameters. Appli¬ 
cation of these data to resource management problems ne¬ 
cessitates development of interpretive methodologies that 
allow quick, consistent, and accurate extraction of pertinent 
information. In the 1960’s, considerable effort was placed 
on developing digital and analog multispectral processing 
capabilities. Most of the capabilities were developed through 
research with aircraft multispectral scanner data. Many ini¬ 
tial image processing capabilities were developed on sys¬ 
tems that provided little flexibility to the analyst, were dif¬ 
ficult to use, and required considerable computer time to 
complete an analysis. However, these capabilities and the 
availability of Landsat data has provided the impetus for 
engineering and development of new, improved, low-cost 
image analysis systems. 1-8 The trend has been toward more 
interactive systems that provide users greater flexibility in 
analyzing multispectral data more rapidly than previously 
possible. 

Evaluation of digital image analysis of Landsat multispec¬ 
tral scanner (MSS) data for mapping and inventorying nat¬ 
ural resources have been made. 9-15 These studies and others 
have led to the development of image analysis techniques 
that are useful in resource inventory and mapping programs. 
The purpose of this paper is to discuss analysis techniques 
that are commonly required for successful application of 
Landsat MSS data to natural resource inventories. 

IMAGE PROCESSING 
Radiometric corrections 

Landsat MSS data are characterized by several types of 
radiometric anomalies including bad data lines and points, 


* Some of the original figures in this paper were in color. Color figures can 
be obtained from the author. 


radiometric striping, and atmospheric scattering effects. 
Corrections or normalization procedures must be used prior 
to image classification to minimize the effect of these 
anomalies on the interpretation and analysis of Landsat data. 

Bad data lines, line segments or pixel dropouts are often 
referred to as intermittent striping problems. Two tech¬ 
niques are commonly used to insert new data in place of bad 
data. One technique is to replace the bad data line with the 
preceding line. A second technique is to replace the bad 
data by interpolating between the brightness values from the 
pixels in the line before and after the bad data line (Table I). 
Although the difference between the two techniques seem 
slight, the latter tends to provide a more accurate represen¬ 
tation of the correct brightness value. 

Radiometric striping is characterized by variations in film 
density that should be uniform but often appear as system¬ 
atic light or dark lines across an image (Figure 1). Radio- 
metric striping introduces anomalous variations in the spec¬ 
tral signatures of resource cover types and will influence 
classification performance because of, (1) high variances, or 
(2) lack of adequate training data to characterize signatures 
of picture elements occurring in lines with striping. This 
striping is caused by unequal response of the detectors in 
the MSS to incident electromagnetic energy. 

The Landsat MSS consists of six detectors for each of 
four spectral bands, a total of 24 detectors. Each of these 
detectors has a slightly different response to the incident 
electromagnetic radiation (EMR) falling upon it, thus, for 
the same intensity of EMR, a slightly different output volt¬ 
age. Specifically, the detectors have different gains and off¬ 
sets (Figure 2). To provide a uniform response, the six 
detectors of each band must be calibrated to a known source 
and corrections applied so that the output of each detector 
is the same for the same value of incident EMR over the 
entire useful range of the detectors. Removal of residual 
differences after calibration constants have been applied is 
called “destriping.” 

Several techniques are available to minimize effects of 
striping. One technique, histogram normalization, calculates 
the mean brightness value for each detector in each band 
(Figure 3). A normalization factor is calculated by ratioing 
the maximum mean to the mean of each detector. The nor¬ 
malization factor for each detector is then multiplied by the 
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Figure 1—Part of a Landsat color composite from scene 5470-19560 acquired on August 1, 1976 over Cantwell, Alaska. Radiometric striping is characterized 
by horizontal lines across the image (see arrow). This image has not been geometrically corrected. 


brightness value of each pixel recorded by the detector and 
the pixel is assigned a new brightness value. Figure 4 shows 
the same image as Figure 1 after destriping has been per¬ 
formed. Although this technique minimizes striping, residual 
striping, caused by localized radiometric anomalies, are 
sometimes found in the data. Other approaches to minimize 
striping effects in Landsat data are described in Reference 
16. 


Geometric rectification 

Landsat digital data have inherent geometric errors that 
must be corrected if the data are to be correctly displayed 
on film, map overlays, or registered to some geographic 
reference system. The geometric errors can be either sys¬ 
tematic and predictable or variable and measurable. 7 Geo¬ 
metric errors that are systematic and predictable can be 
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TABLE I.—Brightness values for each of 12 pixels in 15 consecutive scan 
lines. Line 8, typical of a bad data line, has pixels with brightness values 
considerably higher than pixels from adjacent scan lines. The bad data line 
can be corrected by replacing the data with values from corresponding pixels 
in the preceding line or by interpolating between brightness values from the 
pixels in the line before and after the bad data line. 
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DIFFERENT GAINS AND OFFSETS 
FOR 3 BAND 4 DETECTORS IMAGING 


THE SAME LANDSCAPE CONDITIONS 

Figure 2—Causes of radiometric striping in Landsat data 


modeled and corrections applied to the data. 7,17 However, 
errors caused by variations in spacecraft attitude and alti¬ 
tude are not systematic or predictable. In natural resource 
inventories, accurate location of sample plots or specific 
resource features are required, thus, the non-systematic er¬ 
rors must also be corrected. 

When variable errors are in the data, the effect of the 
errors can be determined by measuring the apparent dis¬ 
placements of ground control points. Ground control points 
are features that can be located in an image and whose 
geographic positions are known. 

Once the displacement of control points is known, cor¬ 
rection functions or mapping transformations are derived 
and used to calculate coordinates and map pixels into their 
correct spatial location on the corrected image. The cor¬ 
rected location of a pixel seldom coincides with a pixel in 
the distorted image, thus the distorted image must be resam¬ 
pled to determine the brightness value of the pixel in the 
corrected image. Three commonly used resampling tech¬ 
niques are nearest neighbor, bi-linear interpolation, and 
cubic convolution. In nearest neighbor resampling, a pixel 
in the corrected image is assigned the brightness value of 
the nearest pixel in the uncorrected image. 

In bi-linear interpolation resampling, the brightness value 
of a pixel in the corrected image is determined by interpo¬ 
lating between brightness values of the four nearest pixels 
in the uncorrected image. In cubic convolution resampling, 
the brightness value of a pixel in the corrected image is 
determined by interpolating between the brightness values 
of 16 nearest pixels in the uncorrected image. More detailed 
discussions of geometric correction and resampling tech¬ 
niques are discussed in References 8 and 17 through 21. 

In most natural resource inventory projects, that have 
utilized Landsat data, correction functions were calculated 
prior to classification, but the actual corrections were ap¬ 
plied to the classification results. This procedure has been 
used largely because it is less expensive to correct one band 
of classification results than to correct four bands of multi- 
spectral scanner data. 

In addition to correcting Landsat MSS data or classifica¬ 
tion results to a geographic reference system, geometric 
correction functions can be used to register one Landsat 
scene to another, register any graphic or map data to Land¬ 
sat data, or compute the location of sample units for re¬ 
source inventory applications. These capabilities are useful 
in subsequent analysis tasks to perform image stratification, 
calculate training statistics, refine image classification, al¬ 
locate sample Units, and estimate the geographic coordinates 
of sample units. 

DIGITAL IMAGE CLASSIFICATION 

Basically, the purpose of digital image classification is to 
group a large number of pixels into a number of meaningful 
categories that can be related to resource features. This 
grouping is normally done by a classification decision rule 
(e.g., minimum distance to the mean, parallelepiped, maxi- 
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mum likelihood). In resource inventory projects, Landsat 
digital data are commonly analyzed using interactive anal¬ 
ysis procedures and a multispectral classification algorithm. 
Most classification algorithms require the analyst to specify 
training statistics for the desired resource classes. 

Developing training statistics 

Fleming, Berkebile, and Hoffer 22 describe three ap¬ 
proaches used to derive training set statistics for use with 
a classification algorithm. These three approaches can be 
referred to as supervised, unsupervised, and modified or 
controlled clustering. When developing training statistics, 
the analyst must ensure that the training statistics represent 
the variation in spectral characteristics of all cover types 
found in the project area. 


In supervised classification, the analyst selects areas that 
are spectrally homogeneous for each resource category. Be¬ 
cause of variations in spectral characteristics of resource 
cover types, there often is more than one training area se¬ 
lected for each category. Training statistics, including the 
mean brightness value within the training area, variance 
about the mean, and the covariance between data channels 
are calculated for each training area. Adaptations of this 
approach use a clustering technique to identify a unique 
number of spectral classes within each training area. In 
either case, the analyst assigns a resource cover type to 
each training area prior to deriving the training statistics. 

Unsupervised development of training statistics uses a 
clustering algorithm to group pixels selected for training into 
a number of relatively homogeneous spectral classes. The 
analyst can usually specify certain parameters to the clus¬ 
tering program including the maximum number of classes, 
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Figure 3—Mean brightness values for each detector in each spectral bank were calculated based on the data from the image shown in Figure 1. Normalization 
factors (K) are determined by ratioing the mean of each detector and the maximum mean for each band. After data are multiplied by the normalization factor, 
the normalized means indicate less variability between detector means, thus minimizing the effect of striping. 






















Digital Image Analysis Techniques 


97 



0 5km 

t—_—__I 

Figure 4—Part of a Landsat color composite from scene 5470-19560 acquired on August 1, 1976 near Cantwell, Alaska. The data have been destriped using a 
histogram normalization technique (compare with Figure 1). This image has not been geometrically corrected. 


maximum standard deviation within a class, and the mini¬ 
mum number of pixels within a class. In addition to the 
clustering parameters, the analyst must also specify the pix¬ 
els to be used in developing the training statistics. He may 
specify all pixels in the project area or randomly select 
sample areas of some specified size. Clustering of all pixels 
in a project area into a small number of spectral clusters or 
classes is sometimes referred to as unsupervised classifica¬ 
tion. After unsupervised classification, it is necessary to 


identify the resource category represented by each cluster. 
An image showing the distribution of the resource categories 
is made by combining all clusters of the same resource 
category into one class. 

When classifying wildland vegetation, it is often difficult 
to identify spectrally homogeneous training areas for each 
resource category. In such situations, clustering of pixels 
within randomly selected training areas is often a useful 
approach for developing training statistics. This approach 
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was used to develop training statistics for classifying wild¬ 
land vegetation and other land cover types on a 118,000 ha 
(293,000 ac) area near Cantwell, Alaska. 

The area shown in Figure 4 was gridded into 4,096 eight- 
pixel-by-eight-pixel cells. Ten percent of the cells were ran¬ 
domly chosen and processed with a clustering algorithm to 
identify a unique number of spectral clusters in the data. 
Fifty-six spectral clusters were identified in the sample data. 
Using this approach to derive training statistics, it was 
assumed that the sample data represented all of the cover 
types in the area. Further, it was assumed that each cover 
type would be represented by at least one spectral cluster 
and that each spectral cluster would represent only one 
cover type. 

A mean brightness value and its variance for each spectral 
band and a covariance matrix was calculated for each spec¬ 
tral cluster. These statistics were used in a maximum like¬ 
lihood algorithm to classify each picture element into one of 
the 56 cluster classes. Color infrared photographs and field 
data collected in July 1976, were used to assign each cluster 
class to one of nine land cover classes. The data were color 
encoded and recorded on film with a film recording device 
(Figure 5). 

Controlled clustering is another approach to developing 
training statistics. Using this approach, the analyst selects, 
either randomly or purposively, polygonal shaped training 
areas that usually include several resource cover types. Pix¬ 
els within the training areas are clustered into a number of 
spectral clusters as in the unsupervised approach. The spec¬ 
tral clusters are printed onto paper or displayed on a color 
monitor. Using aerial photographs, ground data, and other 
supporting data, the analyst identifies the resource cover 
type associated with each spectral cluster prior to perform¬ 
ing the maximum likelihood classification. 22 

Controlled clustering was used to develop training statis¬ 
tics to classify the extent of flooded agricultural land in an 
area near Fargo, North Dakota. Five training areas, each 
approximately 2,200 ha, were located on color infrared pho¬ 
tographs. Areas were selected which represented the range 
of flood conditions and land cover types throughout the 
area. The training areas were located on the Landsat data 
and brightness values of all pixels within the training areas 
were clustered into a large number of spectral clusters. A 
line printer map of each training area was compared with a 
color infrared photographs and each spectral cluster was 
assigned to one of the ground cover classes shown in Table 
II. The statistics for all spectral clusters were used to cal¬ 
culate a statistical measure of the separability between spec¬ 
tral clusters in multidimensional space. The separability sta¬ 
tistic and the interpreted cover class for each spectral cluster 
were used to determine which spectral clusters could be 
combined. The final statistics file contained 21 spectral clus¬ 
ters. 

The training area statistics and the Landsat data were put 
in a maximum likelihood classifier. Evaluation of initial re¬ 
sults indicated confusion between older residential areas and 
agricultural land which had been partly inundated. Fallow 
fields and fields recently plowed and planted were also mis- 
classified as inundated (Figure 6). 


TABLE II.—Land cover classification scheme used in mapping the extent 
of flooding near Fargo, N. Dak. in July 1975. 

Cover type 

1. Urban 

2. Agricultural land not affected by flood 

3. Partially inundated—agricultural land partially affected by flood condition 

4. Completely inundated—agricultural land where effects of flooding were 
apparent over entire field 

5. Standing water 

6. Wooded (both dry and flooded) 


The Landsat digital data were stratified into three strata 
to minimize the variation in land cover and inundated agri¬ 
cultural land within each stratum. A Landsat color compos¬ 
ite was enlarged to a scale of 1:250,000. The general area 
affected by flooding was delineated on the enlarged Landsat 
image and plotted onto U.S. Geological Survey 1:250,000- 
scale maps and digitized. All urban areas greater than ap¬ 
proximately 65 ha were plotted on U.S. Geological Survey 
1:24,000-scale maps, and digitized. The digitized strata 
boundaries were registered to the Landsat image with a 
mean residual error of less than 0.5 pixel. 

The strata boundaries were used to extract the image data 
associated with each strata. The original training statistics 
were modified to reduce the number of training classes and 
minimize the likelihood of misclassification noted in the 
initial results. A separate training statistics file was created 
for each stratum. The training statistics file and appropriate 
Landsat data for each stratum were input to a maximum 
likelihood classification algorithm to classify all picture ele¬ 
ments. After classification, the three strata were added to 
reconstruct the original image. The final classification results 
are shown in Figure 7. Stratification of Landsat data prior 
to classification significantly reduced misclassification error. 

Summarizing classification results 

Image classification results show the areal extent of re¬ 
source cover types. Because the data are in digital format, 
the area within each cover class can be easily tabulated 
(Table III). In most inventory projects, statistical summaries 
are desired by ownership, administrative units within an 
ownership, or political boundaries. If these boundaries are 
plotted onto maps and digitized, the boundaries can be reg¬ 
istered to Landsat (Figure 8). The area of each cover type 
can then be summarized within each defined boundary. This 
technique has been used in several forest and range resource 
inventories. 13 ’ 23-25 Krebs and Hoffer (1976) have also re¬ 
ported on the use of digitized topographic data to summarize 
the area of cover types by elevation, slope, and aspect. 


SAMPLING FOR ACCURACY ASSESSMENT AND 
RESOURCE INVENTORY 

Although areas of cover classes can be tabulated for any 
polygonal area within the Landsat data, there is no indica- 
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Figure 5—Landsat classification of land cover types near Cantwell, Alaska. Barren lands are shown in black. Clear water and sediment-laden rivers are shown 
in dark blue and light blue, respectively. Tundra is shown in dark green, low shrub is shown in light green, and tall shrub is shown in red. Open conifer-tall 
shrub is shown in light red, open conifer-low shrub is shown in violet, and dense conifer is shown in purple. 


tion of classification accuracy nor can a confidence state¬ 
ment be placed on the tabulated figures. In resource inven¬ 
tories, the parameter of interest may not be measured 
precisely on Landsat imagery; thus, sampling procedures 
are needed to derive estimates of the parameter of interest 
and to evaluate classification accuracy. 

Investigators have shown that when remote sensing im¬ 
agery at several scales is available, it is often most efficient 
to first make a large number of fast, inexpensive measure¬ 


ments of a parameter (A*), on small scale, low resolution 
imagery, and then correlate this parameter with the param¬ 
eter of interest (F ; ). 12,13,26,27 A second phase involves se¬ 
lecting a small number of sample units on which the param¬ 
eter of interest (Y t ) is measured precisely on larger scale, 
higher resolution images. Statistical methods are then used 
to estimate the parameter of interest from the measurement 
made on each image type and to estimate the variability 
about the estimated total. 
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TABLE III.—Area Classified into Each Cover Class in the Cantwell, 
Alaska Area Shown in Figure 5. 


Cover Type 

Hectares 

Acres 

Clear water 

1240 

3063 

Sediment laden water 

2575 

6364 

Barren land 

20508 

50672 

Tall shrub 

7671 

18955 

Low shrub 

26769 

66144 

Tundra 

45524 

112488 

Open conifer/low shrub 

3034 

7497 

Open conifer/tall shrub 

8130 

20089 

Dense conifer 

3159 

7805 


118610 

293077 



Figure 6.—Landsat classification results showing agricultural land affected 
by flooding near Fargo, N.D. Wooded areas are shown in light green. Urban 
areas are shown in red, and cropland not affected by flooding in dark green. 
Land where complete inundation occurred is shown in dark blue (standing 
water) and dark brown (land where there was no standing water on July 14, 
1975). Land which was partly affected by flooding is shown in yellow. Initial 
results indicated confusion between older residential areas and agricultural 
land which had been partly inundated (see arrows). 


Verification of classification accuracy 

Evaluation of classification accuracy necessitates sam¬ 
pling each of the cover classes. One approach that can be 
used is a clustered-stratified random sampling technique. 
This technique will provide the user an estimate of the ac¬ 
curacy of cover types displayed on classification map 
overlays. This procedure allows evaluating accuracy on a 
pixel-by-pixel basis and also takes advantage of the effi¬ 
ciency in cluster sampling. To estimate the classification 
accuracy for the area shown in Figure 5 with an allowable 
error of ± 5 percent at the .95 probability level, 2,050 indi¬ 
vidual pixels would have to be sampled. The procedure for 
estimating sample size is presented by Rohde and others. 28 
Because a large number of pixels were to be sampled, cluster 
sampling was used to reduce the amount of time required to 
locate plots and interpret specific pixels. Analysis tech¬ 
niques are used to superimpose a sample unit grid on the 
classification results, randomly select sample units, and tally 
the results within the selected sample units. 

The classified image was gridded into five pixel-by five- 
line grid cells with each grid cell containing 25 pixels (Figure 
9). Each grid cell constitutes a cluster. Sixty-two clusters 
were randomly selected and plotted on aerial photographs 
(Figure 10). Each photo was subdivided into 25 subplots 
(each subplot is equivalent to a pixel). 

The predominant cover type at each subplot was inter¬ 
preted from the aerial photographs. After the photo inter¬ 


pretation was completed, the computer classification of each 
sample unit was tallied and the classification of each subplot 
was compared to the photo interpreted cover type. The 
overall classification accuracy was estimated to be 84.5 
percent ± 4.2 percent at the .95 probability level. An ac¬ 
curacy statement was calculated for each cover class using 
statistics described in Rohde and others. 28 

Sampling procedures for area estimates 

Multispectral classification techniques were used to clas¬ 
sify the area of agricultural land near Fargo, North Dakota 
that was affected by flooding (Figure 7). Although the area 
could be easily tallied, a sampling procedure was required 
to place a confidence interval around the estimate. 



0 llkm 

l... . i 

Figure 7.—Landsat classification results showing agricultural land affected 
by flooding near Fargo, N.D. Image stratification prior to classification re¬ 
duced the misclassification between residential areas and agricultural land 
which had been partly inundated (compare with figure 6). 
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Figure 8.—Part of a Landsat color composite image from scene 2195-17520 acquired on August 5, 1975 (left). Land ownership boundaries were digitized and 
registered to the Landsat image. The Landsat data were classified into three broad volume classes (low, moderate, high) and summarized by ownership. 
Classification results for U.S. Forest Service land is shown on the right. Low volume is shown in white, moderate volume in red, and high volume is shown in 
green. 


A two-phase sampling scheme was used. In two-phase 
sampling, sample units are the same size in each phase. The 
Landsat classification results provided the quick estimates 
in the first phase. A subsample of the first-phase sample 
units was selected for more precise measurements in the 
second phase. A least squares regression of estimates from 
the second phase on estimates from the first phase was used 
to adjust estimates made from the first phase. 

Assuming a sample unit size of 1,600 pixels, the number 
of sample units required for measurement at each phase was 
estimated, based on an expected correlation between meas¬ 
urements made at each phase, the cost ratio of obtaining 
measurements at each phase, and the desired accuracy level. 
To achieve an estimate with an accuracy of ± 10 percent at 
the .95 probability level, it was estimated that 200 first-phase 
samples and 30 second-phase samples would have to be 
measured. 29 A forty-pixel-by-forty-line grid was placed over 
the classification results and two hundred sample units, 
1,600 pixels in size, were randomly selected. The Landsat 
classification results in each selected sample unit were tab¬ 
ulated to provide the first-phase estimates. Thirty second- 
phase sample units were randomly selected from the 200 
first-phase sample units. The latitude and longitude coordi¬ 


nates were computed for the four corner points of each 
second-phase sample unit. Each second-phase sample unit 
was plotted onto U.S. Geological Survey 1:24,000 scaled 
maps and transferred to color infrared photographs. The 
area of cropland affected by flooding in each second-phase 
sample unit was estimated with a dot grid. The area esti¬ 
mates made on the second-phase sample units were used to 
develop regression coefficients to adjust estimates made in 
the first phase from the Landsat classification. 

Using two-phase sampling statistics, 379,165 ha of agri¬ 
cultural land were estimated to be affected by flood condi¬ 
tions. It was also estimated the 148,020 ha were partially 
affected and 236,835 ha were completely flooded. The area 
estimates and associated sampling errors are shown in Table 
IV. 

The ability to overlay a sample unit grid of varying size 
on Landsat data, tabulate classification results within sample 
units, and calculate geographic coordinates of sample units, 
provide data that resource specialists can use to define sam¬ 
pling schemes or allocate sampling efforts to derive resource 
estimates. Landsat data, multi-phase and multi-stage sam¬ 
pling techniques have been used in numerous resource in¬ 
ventory projects. 28 
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Figure 9.—Five pixel by five line sample unit grid superimposed on Landsat classification of land cover types near Cantwell, Alaska. The color scheme is listed 
in figure 2. Sixty-two sample units were randomly selected with equal probability for interpretation on aerial photographs to estimate the classification accuracy. 


OUTPUT PRODUCTS 

Resource managers often need map overlays that show 
the areal extent of resources being inventoried or mapped. 
Common requirements include color encoded map overlays 
or line map overlays at various scales from 1:24,000 to 
1:250,000. Digital classification results provide great flexi¬ 
bility in displaying data in various formats and map scales. 
The data can be easily aggregated for display at various 
levels of generalization or at a given level of detail to sim¬ 
ulate various minimum mapping units. 


In most multispectral classification techniques, each in¬ 
dividual picture element (0.40 ha or 1.1 ac) is assigned to 
one class. This often results in a “salt-and-pepper” appear¬ 
ance on map overlays. Spatial smoothing programs can be 
used to simulate a minimum mapping unit and to reduce the 
“salt-and-pepper” appearance. The image in Figure 11 was 
processed with a smoothing algorithm to simulate an ap¬ 
proximate 4 ha (10 ac) minimum mapping unit. A three pixel 
by three line cell was passed through the data one pixel at 
a time. The first and last lines and the first and last pixels 
in each line were not changed. All other pixels were reas- 
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Figure 10.—Color infrared aerial photograph showing sample unit number 4962. Each sample unit was divided into 25 subplots where each subplot represents a 

pixel (.40 ha or 1.1 ac). 


signed to the most frequently occurring class in the three 
pixel by three line cell. Although this technique tends to 
reduce misclassification caused by edge effects from several 
resource types, small, narrow features may be lost unless 
classes representing such features are more heavily weighted 
when the smoothing function is applied. 

The final classification results can be color encoded on 
film, printed by a line printer, or plotted onto a map overlay 
with a plotter. Color encoded map overlays are made by 
recording onto film the classified digital data that have been 
geometrically corrected, registered and scaled to the desired 
map base. The data can be recorded either directly onto 
color film or onto three separate black-and-white images 
that can be composited into a color image (Figure 11). 

Classification results can be printed with a line printer 
where each cover class is represented by a unique symbol. 


With appropriate geometric correction functions, the data 
can be printed at various scales; the most common are 
1:24,000, 1:62,500, and 1:63,360. 

Classification results can be plotted as polygons on map 
overlays that have been registered and scaled to the desired 
map base. The final map overlay can show each polygon in 
color with different shading patterns or with polygons num¬ 
bered to correspond to a cover class in the legend. Figure 
12 is an example of a map overlay that corresponds to a 
portion of a U.S. Geological Survey 1:63,360 scaled map. 

Other techniques exist to format classification results to 
a user specified grid cell size and register the data to an 
existing information system. This capability allows a re¬ 
source scientist to merge other ancillary data (e.g., soils, 
landforms, hydrology, elevation, slope, aspect) with the 
classification results. 


SUMMARY 


TABLE IV.—Estimate of Agricultural Land Affected by Flood Conditions. 


Cover Class 

Hectares 

Sampling 

Error 



Percent 

Partial inundation 

148,020 

5.4 

Complete inundation 

236,835 

6.1 

All areas affected by flooding 

379,165 

3.9 


Landsat digital data with supporting aerial photographs and 
sampling techniques were used to accurately map wildland 
vegetation and inventory flooded agricultural land. Image 
analysis techniques were required to make radiometric and 
geometric corrections. Techniques were needed to stratify 
the Landsat data prior to classification to reduce the prob¬ 
ability of misclassification. 






104 


National Computer Conference, 1978 


Interactive analysis techniques were used to develop 
training statistics and to evaluate initial classification results. 
The classification results were used in sampling schemes to 
evaluate classification accuracy and to inventory the area 
within each cover type. Analysis techniques were used to 
partition the project areas into sample units, summarize 
classification results within sample units, and to select and 
locate sample units. Image analysis techniques were also 


used to plot the classification results on map overlays that 
were registered to 1:63,360 scaled maps. 

Continued development and improvement of image anal¬ 
ysis techniques are needed to make them more efficient. As 
these techniques are improved, Landsat data, used in its 
proper context with supporting data, and sound sampling 
strategies, can be an effective tool in conducting natural 
resource inventories. 



Figure 11.—Landsat classification of land cover types near Cantwell, Alaska. A spatial smoothing algorithm was used to eliminate the “Salt-and-pepper” 
appearance as shown in figure 5. The smoothing was done to simulate an approximate four ha (ten ac) minimum mapping unit. Barren lands are shown in black. 
Clear water and sediment laden rivers are shown in dark green; low shrub is shown in light green, and tall shrub is shown in red. Open conifer—tall shrub is 
shown in light red; open conifer—low shrub is shown in violet, and dense conifer is shown in purple. 
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Figure 12.—Example of Landsat classification results that have been geometrically corrected and plotted on a map overlay. This overlay is part of the total area 
shown in figure 5. The map overlay was scaled to register to the U.S. Geological Survey 1:63,360 Healy A2 map sheet. Barren land is shown in blue, single¬ 
horizontal lines. Clear water and sediment-laden rivers are shown in dark blue, double-horizontal lines and blue dots, respectively. Tundra is shown in white, low 
shrub is shown as green dots, and tall shrub is shown in red dots. Open conifer—tall shrub is shown as green triangles, open conifer—low shrub is shown as dark 
red, double-vertical lines, and dense conifer is shown as dark red single-vertical lines. 
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INTRODUCTION 

State government agencies are becoming increasingly in¬ 
volved in developing and applying digital image processing 
capabilities to fulfill state responsibilities. Experimental vis¬ 
ual Landsat applications were attempted by many state 
agencies from the launch of Landsat-1 in July, 1972. Digital 
applications, however, were generally not attempted in state 
agencies until the middle of 1974. Since then, many states 
have been involved in investigations and demonstration proj¬ 
ects utilizing digital image processing techniques. Several 
states (e.g., Texas, Georgia, and South Dakota) have estab¬ 
lished in-house operational or quasi-operational digital 
Landsat capabilities. It is clear that the trend of the future 
is toward the establishment of such capabilities in many 
states. 

Often, the development of digital Landsat analysis and 
application capabilities is within the context of a state nat¬ 
ural resource information system. Digital Landsat data is 
extremely compatible with geographic information systems, 
once compensation is made for technical problems. Exis¬ 
tence of a state system provides a focal point for adopting 
satellite remote sensing techniques, thereby facilitating tech¬ 
nology transfer efforts. On the other hand, in states which 
do not have a natural resource information system, the de¬ 
velopment of Landsat digital analysis capabilities has often 
led to the examination of more general computerized natural 
resource data analysis systems. In either case, there is a 
synergistic effect from the mixing and comparing of several 
types of data in a computer. 

The experiences of two state programs in developing or 
acquiring and applying digital Landsat analysis capabilities 
are discussed below. The approach, emphasis and results 
are quite different in each case. All fifty states, however, 
are different in terms of institutional structure, political cli¬ 
mate, role of state government and local needs. While each 
state is unique overall, a number of features and experiences 
are shared by many. By studying similar type developments 
in other states before attempting them at home, pitfalls can 
be avoided and successes repeated. Hopefully, the experi¬ 
ences of North Dakota and South Dakota will prove useful 
in this vein. 


THE NORTH DAKOTA REGIONAL 

ENVIRONMENTAL ASSESSMENT PROGRAM’S 

STATE LAND COVER ANALYSIS 

The North Dakota Regional Environmental Assessment 
Program (REAP), a division of the State Legislative Council, 
was created by the 1975 North Dakota Legislative Assembly 
as a result of a recognized need for a comprehensive and 
coordinated source of natural resource-related information. 
The immediate impetus for such a program was current and 
prospective large-scale coal development in southwestern 
North Dakota. Enabling legislation directed REAP to carry 
out studies and research related to natural resources and to 
develop a comprehensive information and analysis system 
for the storage, retrieval, and analysis of such information. 
Furthermore, REAP was especially directed to develop the 
capability of forecasting the impact of natural resource de¬ 
velopment alternatives which may be considered by the 
state. 

In order to meet its responsibilities, REAP has undertaken 
four specific tasks: (1) the collection of existing baseline 
data, together with a major effort to fill recognized data 
gaps; (2) the design, development, and installation of a com¬ 
puter-based information system capable of storage, re¬ 
trieval, representation, and analysis of appropriate data; (3) 
development of a variety of capabilities, including models, 
for assessing the impacts of proposed natural resource de¬ 
velopment; (4) the establishment of a means for the regular 
monitoring and updating of data so the information system 
will remain current and the analyses relevant. 

In the fall of 1975, REAP conducted an exhaustive study 
of the relevant data sources and gaps. Through this effort, 
it was recognized that there was no comprehensive land 
cover inventory for the state of North Dakota. Accordingly, 
REAP identified the completion of such an inventory as a 
major task requiring early attention. 1 

Several different technologies were consideered for in¬ 
ventorying the land over the entire state of North Dakota, 
approximately 180,000 sq. km. (70,665 sq. mi.) in area. The 
technology chosen, satellite remote sensing, uses imagery 
obtained from the Landsat satellites launched by the Na¬ 
tional Aeronautics and Space Administration in July 1972 
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and January 1975. One of the major advantages of Landsat 
imagery is that the entire state of North Dakota can poten¬ 
tially be covered every nine days, and therefore repetitive 
comparative imagery is available. This technology therefore 
provides a mechanism for conducting a baseline land cover 
analysis for the state and for monitoring how that land cover 
changes over time. Since a near-permanent platform for 
imaging is available at no cost to the state, it is more eco¬ 
nomical to use such a system than to undertake complete 
aerial photographic coverage of the state, and to propose 
frequent repetition of the coverage for monitoring purposes. 
Another distinct advantage is the large areal coverage for 
any given image which permits discerning gross land fea¬ 
tures often not perceivable with aerial photography. Landsat 
imagery is particularly amenable to computer analysis, 
whereas such techniques have not been well developed for 
photographic imagery. A disadvantage of utilizing Landsat 
imagery is that a high degree of detail, particularly in urban- 
type areas, is usually not available. Thus, it is generally not 
feasible to attempt to obtain a full level II land cover analysis 
utilizing Landsat imagery; a thorough level I analysis is 
readily obtainable, however. 2 


Categorization 

Computer processing of the Landsat data from computer 
tapes as received from the EROS Data Center was per¬ 
formed by the Bendix Aerospace Systems Division using a 
Bendix datagrid digitizer system and the Bendix multi-spec¬ 
tral data analysis system (MDAS). 3 The process involved 
identifying an area on a computer display terminal which 
coincided with the exact shape and location of a given train¬ 
ing set. A computer was then coded to identify all areas in 
that particular scene with a similar reflectance. Finally, a 
color was assigned to indicate all such areas. In a follow-up 
process, color-coded areas of a particular category were 
checked against the available ground truthing information to 
ensure that the computer search for similar cover was com¬ 
plete. This process was repeated for each of the land cover 
categories. The final result was what is called a “catego¬ 
rized” tape. 

As a result of categorization of the 19 Landsat scenes, as 
few as 12 and as many as 36 different categorizations were 
determined. For example, in two different scenes up to 14 
different categories of water were identified. As many as 
nine different categories of cropland were detected but there 
were only three different wetland categories. All categories 
identified through this process have been retained on the 
categorized tapes. However, for purposes of displaying the 
results of this land cover analysis on colored maps, an ex¬ 
tensive merging of categories was undertaken. 

Ten different land cover categories were selected for dis¬ 
play on maps and a color was assigned for each category. 
The 10 categories selected for map display include all seven 
applicable Level I categories suggested by the U.S. Geolog¬ 
ical Survey, 2 and five Level II and Level III sub-categories. 


Results 

North Dakota is one of the first states for which detailed 
land cover analysis has been completed utilizing computer 
processing of digital Landsat imagery. The results of this 
land cover analysis have been incorporated in several dif¬ 
ferent products which will be available for public purchase 
and technical use. Each of these products will be described 
in some detail below. 

The first product of the land cover analysis is a color- 
coded map for each of the 53 North Dakota counties at a 
scale of 1:126,720 (exactly two miles to the inch). On each 
map, a color is used to designate each of the 10 different 
land cover categories as indicated in Table I. Each county 
map contains information on the scale, the color legend, the 
name of the county, a small map of the state with the county 
darkened, and an outline of the county containing the date(s) 
of the Landsat imagery used for the analysis. To produce 
these maps Bendix developed a technique to create a high 
quality color negative merging the land cover and the base 
map data. The land cover data was produced by the com¬ 
puter processing discussed above, and the base map was an 
edited version of the detailed county highway maps. 

The second product of this analysis is an area tabulation 
for each township, county, and the state. For each of these 
geographic units, a table was prepared indicating the acreage 
covered by each of the 10 land cover analysis categories. 
The area tabulations were prepared by reading a digital tape 
file and summing the land cover to the appropriate geo¬ 
graphic area. The area tabulation for the entire state is 
shown in Table I. 

The third product of the land cover analysis is a map o? 
the entire state of North Dakota at a 1:500,000 scale (ap¬ 
proximately eight miles to the inch) which contains the land 
cover illustrated by the appropriate color, the same as used 
on the county maps, a legend and the county boundaries. 

The final product of the land cover analysis is a digital 
tape file of land cover data prepared in a specified cell 
format. The objective of this effort was to prepare land 
cover information aggregated to 40-acre cells, registered by 


TABLE I—North Dakota Land Cover by Category 


Land Cover Category 

Acres 

Percent 
of Total 

Built-up Land 

19,777 

0.04 

Cropland 

15,956,690 

35.24 

Fallow Land 

6,189,168 

13.67 

Exposed Subsoil or Saline 



Seep 

100,952 

0.22 

Rangeland 

11,889,387 

26.26 

Mixed Range, Agriculture, and 



Pasture 

7,630,329 

16.85 

Forest 

386,823 

0.85 

Water 

1,265,812 

2.80 

Wetland 

780,284 

1.72 

Barren Land 

975,863 

2.16 

Uncategorized 

80,787 

0.18 

Total 

45,275,872 

99.99 
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quarter-mile section of the Public Land Survey. Thus, all 
pixels with center points lying within the given quarter- 
quarter section were tabulated and grouped by land cover 
category. 

Summary 

The first complete land cover analysis of the state of 
North Dakota has been conducted utilizing computerized 
analysis of Landsat imagery. The products of the analysis 
are color-coded maps of each of the 53 counties in the state 
and a state map. Tables of area coverage of each land cover 
category are also available. The digital information for each 
pixel in the state is recorded on computer tape for subse¬ 
quent utilization in the REAP computer system. 

This analysis has revealed a number of unique capabilities 
of the Landsat imagery process, those utilizing more than 
a single Landsat scene. The products of this analysis are 
already being used in corridor analysis for transmission 
lines, resource analysis, water management problems, non¬ 
point source pollution studies, and regional environmental 
analysis. The results provide an essential data base for the 
REAP system and will serve as the basis fof monitoring 
changes in land cover in the state of North Dakota. 


LANDSAT APPLICATIONS OF THE SOUTH 

DAKOTA LAND RESOURCE INFORMATION 

SYSTEM 

In the early 1970’s, the South Dakota State Legislature 
recognized the need for land related policy and planning 
decisions to be based on solid, objective data. In 1974, the 
Legislature facilitated the establishment of this information 
base by authorizing the South Dakota State Planning Bureau 
to investigate the means for collecting the desired land re¬ 
source data. This investigation resulted in the initiation of 
the South Dakota Land Use Inventory and the formulation 
of the Land Resource Information System (LRIS). The in¬ 
ventory, using both photographic and digital Landsat data, 
began as a three-year, three-phase cooperative project be¬ 
tween the State Planning Bureau and the EROS Data Center. 
The agreement specified that EROS would supply technical 
assistance and imagery, while the state would supply per¬ 
sonnel to initiate and maintain an in-house system for inter¬ 
preting, storing and applying the land use data to South 
Dakota operations. 

One important factor regarding the South Dakota Land 
Use Inventory needs to be emphasized. The State Land Use 
Inventory program established an in-house analysis capa¬ 
bility that will allow mapping as the need arises. The ability 
to perform special manipulations with existing data, the tools 
for updating the inventory, and the capability for special 
projects such as the 208 work make the program especially 
flexible and valuable to the state. Many of the applications 
outlined below have resulted from ad hoc projects that ne¬ 
cessitated some form of additional manipulation of analysis 
of the standard county land use data. 


Phase I—Land use analysis 

The initial phase of the South Dakota Land Use Inventory 
produced a map displaying a generalized picture of the dis¬ 
tribution of major land uses across the state. Seventeen 
Landsat scenes from the spring of 1973 were required to 
provide cloud-free coverage of the entire state. These images 
were manually interpreted to determine the dominant land 
use found within every quarter section (160 acres) in South 
Dakota. The categories classified included urban, rangeland, 
agriculture, forest, water, wetlands, and barren land.* This 
phase of the land use inventory was intended to produce an 
educational tool in the form of a map that displayed the 
complexity of land use across South Dakota. The total pro¬ 
duction time for this map was 24 weeks. The total cost for 
production and dissemination of the map was $4,920, or 6 <t 
per square mile. 

The Phase I state land use map provided an overview of 
the current land use of South Dakota. Due to the crude level 
of detail, the map could not be utilized for detailed planning 
activities. However, it did stimulate much interest in the 
value of Landsat data for large area investigations. While 
most requests were from educational organizations inter¬ 
ested in its academic value, several agencies did utilize the 
map for management purposes. 

Phase II and III land use 

State Planning Bureau anlaysts decided that more detailed 
land use information could only be obtained from Landsat 
data by utilizing more advanced analysis techniques than 
those used to produce the Phase I map. Landsat computer 
compatible tapes, which contain the same information as 
photographic Landsat imagery but in much greater detail 
(1.1 acre cells), provide this additional information. Com¬ 
puter software allowing this analysis was developed during 
fiscal years 1976 and 1977, and is contained in a computer 
package entitled Landsat Imagery Analysis Package 
(LIMAP). 4 These methods were employed early in Phase II 
to initiate a comprehensive, statewide land use inventory. 
The major differences between the output of this inventory 
and the Phase I land use mapping effort are: 

(1) The maps produced in Phases II and III are based on 
data in 1.1 acre cells rather than 160 acre cells. The 
1.1 acre data can be displayed with that cell size or 
aggregated to any meaningful larger cell size such as 
10 or 40 acre cells (Figure 1). 

(2) The classification scheme of Phases II and III contains 
more detailed land use categories. Table II contains 
a complete listing of the categories used in the digital 
land use inventory. 

(3) Phases II and III maps are produced on a county 


* This classification scheme is a Level 1 categorization adopted from: An¬ 
derson, J. R., et al. A Land-Use and Land Cover Classification Scheme for 
use with Remote Sensor Data USGS Professional Paper 964, Washington, 
D.C., 1976. 
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Figure 1—Cow Creek Township, May 15, 1973, data aggregation 
















































































































































































































































































































Digital Image Analysis Applications 


111 


rather than on a statewide basis. The enormous quan¬ 
tity of data required to complete a 1.1 acre inventory 
of South Dakota necessitated storing the data in 
county files. These files, however, can be easily ma¬ 
nipulated to produce maps of many different areas, 
including townships, watersheds, multicounty regions, 
or other regular and irregularly shaped areas. 

(4) The digital format of the Phases II and III data per¬ 
mitted the development of computer software to rap¬ 
idly scale the data to virtually any scale and carto¬ 
graphic projection. This added flexibility satisfies the 
needs of a variety of users by matching the scale of 
the land use data to customized base maps. 

These increases in data content and display capabilities 
greatly enhance the potential applications of the land use 
data to planning and managing South Dakota’s natural re¬ 
sources. 

Current inventory status 

The State Planning Bureau initiated the digital processing 
of Landsat imagery in 1975. As of September, 1977, 50 of 
67 South Dakota counties had been mapped with four ad¬ 
ditional counties in progress. 

The Planning Bureau has recently initiated a program to 
assess the accuracy of the completed portion of the land use 
inventory. Preliminary indications are that the average ac¬ 
curacy of the data is approximately 75 percent. Analysis 
results have progressively yielded higher accuracies with 
the most recently mapped counties being as accurate as 96 
percent. Recognizing the importance of accurate land use 
data to decision makers, the State Planning Bureau has 
adopted a policy of only disseminating maps of at least 85 
percent accuracy. 

Costs 

The time and cost requirements that must be met if the 
inventory is to be economically viable are critical. The State 


TABLE II—South Dakota Landsat Land Use Classification Scheme 

Level I Level II Level III 

100 Urban 
200 Agriculture 

210 Cropland 

211 Bare soil 
213 Small Grains 
215 Large Grains 

300 Rangeland 
400 Forestland 

410 Deciduous 
420 Evergeeen 

500 Water 
600 Wetlands 


Planning Bureau has committed approximately five man- 
years and $300,000 toward the completion of the South Da¬ 
kota Land Use Inventory. This includes a large amount of 
time and money devoted to computer software development. 
While this has increased both time and cost expenditures, 
it has allowed the formulation of an analysis system 
equipped to meet South Dakota’s specific needs. 

A detailed look at current analysis costs, excluding over¬ 
head, training and development expenses, shows the eco¬ 
nomic feasibility of recently produced land use maps. Ana¬ 
lysts have been able to map a 3320 square mile area in 
northeastern South Dakota (approximately 4 counties) for 
a total analysis cost of $5,703.36. The Landsat data was 
mapped using four categories: water, large grains, small 
grains and pasture. The level of detail was 1.1 acres and the 
cost per square mile was $1.72 or $0.0027 per acre. Table 
III contains a detailed listing of mapping costs. These figures 
will not necessarily apply directly to other geographic areas, 
but can be considered fairly typical. 

Applications 

The true value of any land inventory is best determined 
by its applicability to management and policy problems. In 
addition to the South Dakota applications mentioned in 
Table V, the State Planning Bureau has provided publica¬ 
tions and advice to 40 states and four foreign countries. 
While this indicates wide interest in the innovative nature 
of the South Dakota Land Use Inventory and the Land 
Resource Information System, it is the state applications 
that best illustrate the overall value of the program. 

Limitations of Landsat data 

While Landsat has generally been adequate as the primary 
data source for the land use inventory, it does have limita¬ 
tions. Scanner sensitivity and resolution are sometimes in- 


TABLE III—Time and Cost Data for Digital Land Use Mapping of 
a 3320 Square Mile Area in Northeastern South Dakota. 


TIME 


Data Rectification 


2 hours 

Classification 


21 hours 

Verification (Accuracy) 


68 hours 

Final Map Production 


18 hours 

COSTS 

Landsat Imagery 

1 CCT @ $200 

$ 

109 hours 

200.00 

Computer Time 

7.182 hours @ $625.00/hr. 

$4,488.75 

Salary 

109 hours @ $6.94/hr. 

$ 

754.61 

Ground Data Acquisition 

4 counties @ $40.00/county 

$ 

160.00 

Misc. Supplies (Paper, tapes, etc.) 

$ 

100.00 


700 Barren Land 


630 Riparian Vegetation 


$5,703.36 
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adequate for the task of mapping certain land use categories. 
Wetlands, for instance, have an appearance similar to large 
grains on Landsat data obtained during the summer months. 
This often results in the misclassification of wetlands. While 
techniques exist to correct this problem, they increase both 
analysis time and cost. Sensor resolution limitations pre¬ 
clude the use of Landsat for meaningful classification of the 
state’s urban areas. Again, alternative inventory methods 
such as aerial photography are available but at an increased 
analysis cost. 

Another limitation of Landsat data is its timeliness. Raw 
data in the form of computer compatible tapes is currently 
available between three and four months after the satellite 
has recorded the scene. This data turnaround is often in¬ 
adequate for ad hoc analysis since categorization of the data 
can require up to an additional month. Landsat can be a 
timely tool for monitoring natural disasters such as floods 
and droughts. However, to be of use, the data should be 
available within one to three weeks. When this data avail¬ 
ability problem is solved, the utility of Landsat data to South 
Dakota will be greatly enhanced. 

When the limitations of the Landsat data are recognized, 
satisfactory results can be easily and consistently obtained. 
The South Dakota State Planning Bureau has generally been 
satisfied with the results of its statewide digital land use 
inventory. The Bureau plans to complete the land use in¬ 
ventory during fiscal year 1978, and then to publish a state 
land use atlas containing the resultant data. Because of the 
successes achieved so far, the State Planning Bureau intends 
to continue using Landsat as an important tool in the plan¬ 
ning and decision-making process. 

IMPLICATIONS TO THE COMPUTER WORLD 

The accelerating development of state digital image proc¬ 
essing systems and natural resource information systems 
will provide many challenges for the computer world over 
the next decade. A great deal of development will need to 
be completed on both hardware and software. 

Multivariate statistical analysis of image data requires a 
great deal of computing power. As future image data sets 


TABLE IV—Typical Landsat Applications of the South Dakota Land 
Resource Information System ' 

Comprehensive Planning 
County Comprehensive Plans 
U.S. Army Corps of Engineers Watershed Planning 
State and Regional River Basin Planning 
HUD 701 Land Use Planning 
Environmental Impact Assessment 
Energy development impact analysis 
Transportation corridor selection 
Energy transmission corridor analysis 
Surface Water Inventory 
Impoundment location 
Flood plain delineation 
Land Use Change Analysis 
Agricultural conversion 
Urbanization 


grow an order of magnitude in size and additional channels 
are added to future sensors, much faster processors will be 
required. Mass storage devices capable of storing hundreds 
of megabits of data with access times comparable to main 
storage need to be developed for specialized processing ap¬ 
plications such as data rectification and image enhancement. 
Video image display devices will have to be upgraded to 
handle larger volumes of data. Remote data transmission 
devices will have to be accelerated as distributed image 
processing systems become more prevalent and transmis¬ 
sion requirements are greatly increased. Color hard copy 
output devices will need to be improved to upgrade product 
quality and reduce production costs. Above all, a great deal 
of system’s integration work will be required to effectively 
and efficiently pull together all the various hardware com¬ 
ponents required for operational image processing applica¬ 
tions. 

On the software side, a great deal of development can be 
expected. Generalized applications software packages will 
need to be prepared for a wide variety of mainframes and 
minicomputers as more and more organizations adopt re¬ 
mote sensing technology. Much custom software develop¬ 
ment will be required to tailor such packages to meet indi¬ 
vidual user needs. Interactive systems will become more 
prevalent as user sophistication increases. 

Overall, development of image processing and information 
system capabilities provides the computer industry with an 
important opportunity to contribute to improving the quality 
of life and the wisdom of resource management in America 
and the world. While land and resource shortages have not 
yet reached crisis proportions, world population growth is 
bringing that day increasingly closer. New technologies 
alone will not solve these problems. Hopefully they will 
prove to be useful tools to the people who must. 
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In perspective—Meeting the image processing 
challenge for remote sensing 

by PHILIP H. SWAIN 

Purdue University 
West Lafayette, Indiana 


INTRODUCTION 

Although man has been observing the condition of his en¬ 
vironment from aerial vantage points for many decades, 1 
developments of the 1950s and 1960s could be pointed to as 
the origins of modern remote sensing technology. 2 The last 
twenty to thirty years have seen dramatic advancements in 
the design of sensor systems, particularly in the measure¬ 
ment of infrared energy, the advent of the digital computer, 
and progress in modeling some of the processes associated 
with human intelligence, notably pattern recognition. Within 
this decade we have developed the capability to marry these 
advancements with spacebome systems which can provide 
us image data of the earth—and other planets as well—of 
unparalleled scope, scale, resolution and timeliness. The 
“dimensions” of the data, however, overwhelm our human 
capabilities to assess and assimilate it. The human visual 
system cannot deal effectively with the four or thirteen or 
more than twenty concurrent images of the same scene 
produced by modern multispectral imaging systems. Nor 
can we begin to cope with the volume of image data that 
has suddenly become available through the orbiting of such 
systems. 

And thus the challenge: to develop effective computerized 
image processing techniques, either to reduce the flood of 
data to proportions we can handle in a useful way or to 
amplify our powers to manipulate the data and discern the 
useful information contained in it. How are we meeting this 
challenge? First, let us look a bit more closely at what is 
involved. 


UNIQUE ASPECTS OF THE PROBLEM 

In remote sensing, information about the earth and its 
atmosphere is conveyed to the sensor system through the 
spatial, spectral and temporal variations of the electromag¬ 
netic energy emanating from the scene. 2 Typically we or¬ 
ganize the spatial variations into an image which can then 
provide a two-dimensional characterization of “things” in 
the scene in terms of their shape, size, orientation, and 
context. 


To a limited extent, the spectral or wavelength-dependent 
variations in the scene can also be incorporated into an 
image through the dimension of color. However, these var¬ 
iations can be quantified to only a limited extent in this way 
because discernible “color” is a net effect which is uniquely 
definable by at most three fundamental components. Thus 
since our modem sensor systems can precisely measure 
more than three distinct spectral components we are bound 
to suffer loss of data (and usually, but not always, of infor¬ 
mation) if we attempt to represent these components in 
terms of color. An alternative is to represent the various 
spectral components as multiple images rendered in gray 
tones. Unfortunately such a representation is not only awk¬ 
ward but fails to display explicitly the subtle relationships 
between spectral components which may be essential for 
characterizing different ground covers. 

Attempting visual display of temporal variations simply 
makes matters worse. Assuming that images of a given scene 
collected at different times can be precisely registered (the 
technology for accomplishing this has in fact been devel¬ 
oped 3 ), color renditions and multiple gray-tone images can 
again be used to convey temporal variations in the scene. 
But taking these variations together with the key spectral 
variations leads to a level of complexity guaranteed to 
overburden the human sensory system. 

Thus, a characterizing feature of remote sensing data is 
its multi-image nature. Roughly speaking, a computer is as 
capable of dealing with four or thirteen or twenty dimen¬ 
sional data as it is with three dimensional data. 4 So it is 
natural to turn to the computer for assistance in making use 
of remote sensing data. 

We have already mentioned the volume of remote sensing 
data that is available. Even early airborne systems (circa 
1960s) typically produced “scenes” consisting of 10 6 mul¬ 
tispectral pixels (“picture elements”). Today we often wish 
to process ten times that many pixels as a single unit. By 
the early 1980s, improvements in sensor resolution will in¬ 
crease the figure by another order of magnitude. 5 

Data volume is therefore another characterizing aspect of 
remote sensing imagery. We need computers to catalog as 
well as to analyze very rapidly the massive quantities of 
available data. But even with computers serving in these 
capacities, the situation is characteristically more demand- 
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ing in the case of remote sensing than in any other appli¬ 
cation of image processing which comes to mind. 

Remote sensing is very much an applied technology. His¬ 
torically its development has been and continues to be most 
rapid when a potential application provides a focus. As a 
result, image processing for remote sensing has character¬ 
istically been a multidisciplinary effort, resulting in a need 
to make scientists who may not be computer-oriented feel 
“at home” with the image processing machinery. This has 
been accomplished by paying serious attention to the man/ 
machine and man/data interfaces, developing, for example, 
English-like command languages to be used for directing the 
image processing operations. 6,7 

Having sketched some characteristic aspects of image 
processing for remote sensing, we shall now turn to an 
overview of some of the efforts which have been directed 
toward meeting the challenge, what is happening today, and 
some apparent needs for the future. In doing so, it is helpful 
to think in terms of fairly distinct image processing activities: 
enhancement, preceding either visual or machine-imple¬ 
mented analysis; analysis, typically but not exclusively clas¬ 
sification; and product formating, storage and retrieval. 

WHERE HAVE WE BEEN? 

Attention began to focus on the computer analysis of 
multispectral remote sensing data in the mid-1960s. 8 To limit 
the task to one of manageable proportions, it was decided 
to concentrate in the main on the spectral variations in the 
data, practically ignoring, for the time being, the spatial and 
temporal variations. This choice was largely motivated by 
the wealth of information that the spectral domain was as¬ 
sumed to contain and optimism with respect to how much 
effort would be required to extract a significant proportion 
of that information. 

Much of the earliest multispectral image data was re¬ 
corded in analog form, either photographically on film or 
electronically on magnetic tape. Once it had been decided 
to use digital computers to analyze the data, techniques and 
hardware were needed to convert this data from analog to 
digital form and store it on computer-compatible tape. But 
of course the data analyst then needed a facility for display¬ 
ing the digital data and referencing specific locations in it. 
These combined needs gave great impetus to the develop¬ 
ment of image digitizing and digital display devices. In es¬ 
sence, the skills and theories which had been developed for 
the photographic darkroom had to be translated into a new 
digital technology. 

Statistical pattern recognition was quickly adopted as a 
promising approach for analyzing the multivariate data as¬ 
sociated with each pixel in the multispectral image. 9 One of 
the earliest applications was crop species mapping. Com¬ 
mensurate with the relative complexity of this task (some of 
the spectral differences among crops in the field are fairly 
subtle), maximum likelihood classification assuming multi¬ 
variate Gaussian data distributions was often used to accom¬ 
plish the analysis. 4 Interestingly, although many alternative 
classification methods have been investigated since those 


early days, the Gaussian maximum likelihood classifier has 
remained the most widely used. Some reasons which ac¬ 
count for this include its ease of implementation, its mod¬ 
erate level of computational complexity and memory re¬ 
quirements, and its ability to represent moderately complex 
decision surfaces in the feature space through use of second 
order statistics (class covariance matrices as well as means). 

The products of the data analysis were largely in two 
forms: classification “maps” in raster format analogous to 
the data input, usually printed on conventional computer 
lineprinters using distinct characters to represent the ground 
cover classes; and tabular summaries listing total area cov¬ 
erage by class and, for selected areas for which adequate 
ancillary data were available, an estimate of classification 
accuracy. These products were fine for illustrating the kinds 
of information which could be extracted from remote sensing 
data, but they would eventually be found to fall far short of 
the needs of the potential users. The map makers needed 
photo-quality images with tightly controlled geometric char¬ 
acteristics and somewhat less detail than the computer anal¬ 
yses generally produced; and in color. The policy makers 
needed results reported according to specialized categories 
not always identifiable by their spectral properties alone; 
and on the basis of oddly shaped political or jurisdictional 
regions. Once the need for these types of products was 
realized, vigorous work toward their generation met with 
success. 10 

Most of the remote sensing data processing capabilities 
were initially implemented on general purpose computers. 
Although ideal for research and development purposes, 
these implementations were clearly inadequate for the high- 
volume processing which would be called for when satellite- 
borne remote sensor systems were orbited in the early 1970s. 
A few research organizations and commercial establish¬ 
ments attempted to predict the image processing capabilities 
which would be needed and to construct special purpose 
systems to meet the anticipated need. Although they suc¬ 
ceeded in demonstrating that special purpose systems could 
achieve dramatically improved throughput performance 
compared to general purpose versions they also proved that 
the remote sensing image processing technology was ad¬ 
vancing at a great rate: their systems were obsolescent even 
before producing their first results. 

WHERE ARE WE TODAY? 

Present-day image processing technology has gone be¬ 
yond the spectral domain alone, making significant strides 
into the spatial domain and some tentative steps into the 
temporal domain. 8 

A number of cosmetic enhancements are routinely applied 
to the digital multispectral data. 11-16 These include a wide 
range of false color display techniques, geometric adjust¬ 
ments to match existing map products, edge enhancement, 
resolution enhancement (reduction of blurring due to various 
sensor system characteristics), and other filtering opera¬ 
tions. 17 Methods have been developed for precisely regis¬ 
tering multiple images of the same scene to a common tern- 
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plate scene. 3,18,19 All of these advances have improved the 
interpretability of remote sensing image data and increased 
the potential utility of the final product to the user. 

Methods for extracting information from remote sensing 
image data have developed to the point where they are 
testing the boundaries of what can be accomplished based 
on the spectral domain alone. With respect to some potential 
applications, the results achieved with data available from 
current sensor systems, of relatively limited spectral cov¬ 
erage and course spectral resolution, have fallen short of the 
results anticipated. This has had the unfortunate effect of 
discouraging segments of the potential user community and 
causing them and many interested layman to prematurely 
judge the limitations of the technology. 

Texture and spatial contiguity are two geometric charac¬ 
teristics of the image data which have been demonstrated to 
be useful in the analysis process. Texture characterizes 
some ground covers of interest; the challenge has been to 
find effective measures of observable texture. 20-22 Detection 
of spatial contiguity permits scene segmentation and the 
delineation and classification of “objects” in the data. 23 
When a large number of “objects” consist of many pixels, 
this leads to greater classification efficiency and accuracy. 
Statistical “sample classifiers” have been developed for this 
purpose. 23 

The products of the image processing operations today 
are a far cry from those available even a few years ago. 
Computer-driven precision film writers, ink-jet and electro¬ 
static printers have made possible excellent quality map-like 
products at increasingly palatable prices. Capabilities for 
digitizing arbitrarily shaped polygonal boundaries have made 
it possible to report quantitative areal tabulations on a very 
flexible basis. 10 For applications where areal estimates are 
more feasible than “wall-to-wall” inventories, sophisticated 
statistical design and evaluation methods have been devel¬ 
oped. 24,25 

Although they are exceedingly expensive, a few relatively 
powerful image processing systems are available for remote 
sensing data processing. 26 In the development of these sys¬ 
tems, great attention has been paid to facilitating the role of 
the system operator who, in fact, is assumed to be a very 
capable data analyst. Thus an important element of the sys¬ 
tem is an interactive high-resolution digital display with a 
color CRT. A programmable array processor rather than a 
hard-wired special purpose processor may be used to imple¬ 
ment the compute-bound steps in the processing. The pro¬ 
cesses for determining the appropriate parameters for var¬ 
ious stages of the processing still involve the data analyst 
and often require considerable trial-and-error, however. As 
a result, the throughput of these specialized systems remains 
rather limited. 


THE NEEDS OF THE FUTURE 

We know from the performance of skilled image inter¬ 
preters faced with visual analysis of available remote sensing 
data that there remains a wealth of information in the spatial 
and temporal domains as yet untapped by computer-imple¬ 


mented algorithms. Effectively characterizing and extracting 
this information constitute the challenge for the future. 

A number of image enhancements appear feasible but 
have yet to be developed sufficiently for routine application. 
Examples include haze removal, scene-to-scene tonal nor¬ 
malization, and boundary detection and enhancement. The 
utility of the data and the task of the data analyst would 
both be improved by development of effective methods for 
coordinating ancillary data with the associated image data. 
This is not a trivial matter when one considers the spectrum 
of data types and formats that can be involved. It is not at 
all clear how to most effectively coordinate point, polygon, 
and line data with image data. 

With precisely registered multitemporal remote sensing 
data promising to become more readily available, what 
means can be developed to effectively display the data and 
to enhance features of interest (often changes) to the image 
interpreter or the data analyst? What new problems will we 
have to deal with due simply to the gigantic proportions of 
the data sets? (Current Landsat scan lines consist of just 
over 3200 pixels, each of which is 4-dimensional. Projected 
systems will yield over 6000 6-dimensional pixels per scan 
line 5 ). 

Judging from the evolving image processing research, quite 
a range of new approaches for characterizing and extracting 
spatial information will be tested on remote sensing data and 
appropriately adapted. Texture, a complex visual phenom¬ 
enon, is one of the features which seems quite obviously 
useful to human photointerpreters. Many very different ap¬ 
proaches to the quantitative characterization of texture have 
been attempted and, in some instances involving remote 
sensing data, have been moderately successful. More prog¬ 
ress in this area is needed. 21 

Where texture is a rather local spatial phenomenon, con¬ 
text might be thought of as a property bridging the local on 
one hand and the global on the other. Context is another 
phenomenon used effectively in manual photo-interpreta¬ 
tion, since it can be quite helpful in characterizing a pixel or 
an object to take into consideration the nature of its neigh¬ 
bors. It has been demonstrated that the capacity to do this 
can be incorporated in pattern recognition algorithms for 
remote sensing, although the computational cost is substan¬ 
tial. 27 

Global relationships in images can be characterized using 
syntactic pattern recognition techniques. 28,29 This approach 
to remote sensing image analysis is very important because 
there are many instances in which rather general spatial 
relationships are key identifying characteristics. Rivers can 
be discriminated from lakes because they are “string-like” 
rather than “blob-like”; clouds can be discriminated from 
snow because clouds cast shadows. To use this approach, 
however, requires the abilities to infer the characterizing 
relationships from typical imagery and to capture them ef¬ 
fectively in “pattern grammars”. These are very difficult 
problems of image analysis in general and much remains to 
be done before practical applications of syntactic pattern 
recognition will be seen in remote sensing. 

The decision processes needed for remote sensing data 
analysis can often be cast rather effectively as hierarchical 
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and/or sequential processes. 30 One can envision, for exam¬ 
ple, classifying a pixel into successively more specific cat¬ 
egories by selectively using additional spectral bands or 
other features. The features to be used could be adaptively 
determined by the decisions made at each stage. This sort 
of hierarchical process can lead to both faster and more 
accurate classification. Still another classifier model can be 
used to efficiently process multitemporal image data from 
successive passes of the sensor over the scene. 31 In this 
case, likelihood computations are “cascaded” and the re¬ 
sults of each stage of computation are made available to the 
next stage when later data is obtained. 

Early in the history of the development of image process¬ 
ing for remote sensing, the goal of total automation of the 
process was established. It was assumed that only through 
total automation could the throughput of the processing sys¬ 
tem be made adequate to the data volume and demand for 
processing. Now, more than a decade later, that goal still 
remains the ideal, but it is widely recognized that it may be 
some time before we learn how to make computers perform 
certain complex tasks that humans perform rather well. Ex¬ 
perience has demonstrated that, at least for the present, we 
cannot devise streamlined automated analysis algorithms 
employing available technology which can serve as a rea¬ 
sonable replacement for the data analyst who has training 
related to the application of interest and the computerized 
analysis tools as well. For the foreseeable future, then, we 
can expect to see continued research to improve the effec¬ 
tiveness of the human data analyst as part of the total image 
processing system. We have already noted the need to better 
coordinate the various forms of data the analyst must use 
(point, line and polygon data as well as the image data). 
More effective means of interacting with the total data bank 
are needed, utilizing, in all likelihood, interactive display 
facilities with graphics, image overlay, and color image ca¬ 
pabilities. Similar remarks apply to interaction with inter¬ 
mediate and final processing results so that the analyst can 
function in a feedback loop to iteratively improve the results 
obtained. 

Most beneficial use will be made of both the sensor data 
and the image processing facilities when a common set of 
these resources can be made available to a wide range of 
potential users. One way to accomplish this is to store fairly 
detailed analysis results in a flexible “earth resources infor¬ 
mation system” capable of being interrogated and providing 
a wide range of graphical and statistical information. This 
can be thought of as the “users’ data base”, containing 
more “refined” information developed by applying image 
processing to the “remote sensing data base”. Some efforts 
have already been made along these lines, 32 * 33 but the needs 
of the user community are not yet being widely met. 

Finally, the still rapidly evolving computer technology— 
microprocessors and parallel and pipeline computation— 
have important implications for the future of remote sensing 
image processing. The sheer volume of the data to be pro¬ 
cessed has already provided motivation for implementing rel¬ 
atively conventional analysis methods (multispectral clus¬ 
tering and maximum likelihood classification) on the 
ILLIAC IV parallel processor. Demonstration runs have 


resulted in computational speed-up factors of two to three 
orders of magnitude (between 10 2 and 10 3 ). 34 Yet this rep¬ 
resents a rather “general purpose” implementation using 
obsolescent technology. The near future will see availability 
of very high speed microprogrammable processors which 
can be dynamically organized into parallel/pipeline config¬ 
urations potentially capable of performing staggering num¬ 
bers of computations per second. 35 It will be no small chal¬ 
lenge to utilize such computational power effectively. 
Significantly, the prospect of having such power available 
is allowing us to think about image processing in new ways, 
to consider developing computational methods which were 
infeasible for practical applications when conventional serial 
implementations had to be assumed. The programability of 
these advanced systems will at once ensure their general 
applicability and make them available for further evolving 
the remote sensing image processing technology. 


REFERENCES 

1. Fischer, W. A., “History of Remote Sensing,” in R. G. Reeves, ed., 
Manual of Remote Sensing, American Society of Photogrammetry, Falls 
Church, Va., 1975. 

2. Landgrebe, D. A., “The Quantitative Approach: Concept and Ration¬ 
ale,” in P. H. Swain and S. M. Davis, eds., Remote Sensing: The Quan¬ 
titative Approach, McGraw-Hill International Book Co., New York, 
1978. 

3. Anuta, P. E., “Spatial Registration of Multispectral and Multitemporal 
Digital Imagery Using Fast Fourier Transform Techniques, IEEE Trans. 
Geoscience Electronics, Vol. GE-8, No. 4, pp. 353-368, 1970. 

4. Swain, P. H., “Fundamentals of Pattern Recognition in Remote Sens¬ 
ing,” in P. H. Swain and S. M. Davis, eds., Remote Sensing: The 
Quantitative Approach, McGraw-Hill International Book Co., New 
York, Germany, 1978. 

5. Hamage, J. and D. Landgrebe, eds., “Landsat-D Thematic Mapper Tech¬ 
nical Working Group Final Report,” NASA Johnson Space Center, 
Houston, Texas, 1975. 

6. Phillips, T. L., ed., LARSYS Version 3 User’s Manual, Laboratory for 
Applications of Remote Sensing, Purdue University, West Lafayette, 
Indiana, 1973. 

7. Swain, P. and D. Germann, “On the Application of Man-Machine Com¬ 
puting Systems to Problems of Remote Sensing,” Software Age, Vol. 2, 
August 1968, pp. 13-20. 

8. Landgrebe, D., “Computer-Based Remote Sensing Technology—A Look 
to the Future,” Remote Sensing of Environment, Vol. 5, 1976, pp. 229- 
246. 

9. Fu, K. S., D. A. Landgrebe, and T. L. Phillips, “Information Processing 
of Remotely Sensed Agricultural Data,” Proc. IEEE, Vol. 57, April 1969, 
pp. 639-653. 

10. Swain, P. H., “Advancements in Machine-Assisted Analysis of Multi¬ 
spectral Data for Land-Use Applications,” Proc. Symposium on Machine 
Processing of Remotely Sensed Data, Purdue University, IEEE Cat. No. 
77CH 1218-7 MPRSD, 1977, pp. 336-343. 

11. Anuta, P. E., “Geometric Correction of ERTS-1 Digital Multispectral 
Scanner Data,” LARS Information Note 103073, Laboratory for Appli¬ 
cations of Remote Sensing, Purdue University, West Lafayette, Indiana, 
1973. 

12. Anuta, P. E., “Spline Function Approximation Techniques for Image 
Geometric Distortion Representation,” LARS Information Note 103174, 
Laboratory for Applications of Remote Sensing, Purdue University, West 
Lafayette, Indiana, 1974. 

13. Crane, R. B., “Preprocessing Techniques to Remove Atmospheric and 
Sensor Variability in Multispectral Scanner Data,” Proc. Seventh Inter¬ 
national Symposium on Remote Sensing of Environment, Environmental 
Research Institute of Michigan, Ann Arbor, Michigan, 1971, pp. 1345- 
1355. 



Meeting the Image Processing Challenge for Remote Sensing 


117 


14. Emmert, R. A., and C. D. McGillem, “Multitemporal Geometric Distor¬ 
tion Correction Using the Affine Transformation,” Proc. Symposium on 
Machine Processing of Remotely Sensed Data, Purdue University, IEEE 
Cat. No. 73 CHO 834-2 GE, 1973, pp. IB-24 to IB-32. 

15. Higgins, J. L., and E. S. Deutsch, “The Effects of Picture Operations in 
the Fourier Domain and Vice Versa,” in F. Shahroki, ed.. Remote Sens¬ 
ing of Earth Resources, Vol. 1, University of Tennessee, Tullahoma, 
Tennessee, 1972, pp. 460-480. 

16. Ulmer, D. E., “Processing and Enhancement of Landsat Imagery,” Proc. 
IEEE Midcon/77, November 1977. 

17. Lintz, J., Jr., and D. S. Simonett, eds., Remote Sensing of Environment, 
Addison-Wesley, Reading, Massachusetts, 1976. 

18. Bamea, D. K., and H. F. Silverman, “A Class of Algorithms for Fast 
Digital Image Registration,” IEEE Trans. Computers, Vol. C-21, No. 2, 
February 1972, pp. 179-186. 

19. Lillestrand, R. L., “Techniques for Change Detection,” IEEE Trans. 
Computers, Vol. C-21, No. 7, July 1972, pp. 654-660. 

20. Haralick, R. M., K. Shanmugam, and I. Dinstein, “Textural Features for 
Image Classification,” IEEE Trans. Systems, Man, and Cybernetics, Vol. 
SMC-3, November 1973, pp. 610-621. 

21. Weszka, J. S., C. R. Dyer, and A. Rosenfeld, “A Comparative Study of 
Texture Measures and Terrain Classification,” IEEE Trans. Systems, 
Man and Cybernetics, Vol. SMC-6, April 1976, pp. 269-285. 

22. Carlton, S. G., and O. R. Mitchell, “Image Segmentation Using Texture 
and Gray Level,” Proc. IEEE Computer Society Conference on Pattern 
Recognition and Image Processing, Rensselaer Polytechnic Institute, 
IEEE Cat. No. 77CH1208-9 C, 1977, pp. 387-391. 

23. Kettig, R. L., and D. A. Landgrebe, “Classification of Multispectral 
Image Data by Extraction and Classification of Homogeneous Objects,” 
IEEE Trans. Geoscience Electronics, Vol. GE-14, No. 1, January 1976, 
pp. 19-26. 

24. Wigton, W. H., “Use of LANDSAT Technology by Statistical Reporting 
Service,” Proc. Symposium on Machine Processing of Remotely Sensed 
Data, Purdue University, IEEE Cat. No. 76 CH 1103-1 MPRSD, 1976, 
pp. PB-6 to PB-10. 

25. Bauer, M. E., M. M. Hixson, B. J. Davis, and J. B. Etheridge, “Crop 
Identification and Area Estimation by Computer-Aided Analysis of Land¬ 
sat Data,” Proc. Symposium on Machine Processing of Remotely Sensed 


Data, Purdue University, IEEE Cat. No. 77CH 1218-7 MPRSD, 1977, 

pp. 102-112. 

26. Carter, V. P., F. Billingsley, and J. Lamar, Summary Tables for Selected 
Digital Image Processing Systems, Open File Report 77-414, U.S. Geo¬ 
logical Survey, Reston, Va., May 1977. 

27. Welch, J. R., and K. G. Salter, “A Context Algorithm for Pattern Rec¬ 
ognition and Image Interpretation,” IEEE Trans. Systems, Man and 
Cybernetics, Vol. SMC-1, January 1971, pp. 24-30. 

28. Fu, K. S., Syntactic Methods in Pattern Recognition, Academic Press, 
1974. 

29. Brayer, J. M. and K. S. Fu, “Application of a Web Grammar Model to 
an Earth Resources Satellite Picture,” Proc. Third International Joint 
Conference on Pattern Recognition, IEEE Cat. No. 76CH1140-3 C, 1976, 
pp. 405-410. 

30. Swain, P. H. and H. Hauska, “The Decision Tree Classifier: Design and 
Potential,” IEEE Trans. Geoscience Electronics, Vol. GE-15, July 1977, 
pp. 142-147. 

31. Swain, P. H., “A Versatile Classifier Model for Multiobservational Anal¬ 
ysis” (abstract only), Proc. Symposium on Machine Processing of Re¬ 
motely Sensed Data, Purdue University, IEEE Cat. No. 77CH 1218-7 
MPRSD, 1977, p. 307. 

32. Bryant, N. A., and A. L. Zobrist, “IBIS: A Geographic Information 
System Based on Digital Image Processing and Image Raster Datatype,” 
Proc. Symposium on Machine Processing of Remotely Sensed Data, 
Purdue University, IEEE Cat. No. 76 CH 1103-1 MPRSD, 1976, pp. 
1A-1 to 1A-7. 

33. Hicks, J. E. and T. Hauger, “Managing Natural Resource Data: Min¬ 
nesota Land Management Information System,” Council of State Gov¬ 
ernments, Lexington, Ky., May 1977. 

34. Ray, R. M., Ill, and H. F. Huddleston, “Illinois Crop-Acreage Estima¬ 
tion Experiment,” Proc. Symposium on Machine Processing of Remotely 
Sensed Data, Purdue University, IEEE Cat. No. 76 CH 1103-1 MPRSD, 
1976, pp. PB-14 to PB-21. 

35. Allen, G. R., L. O. Bonrud, J. J. Cosgrove, and R. M. Stone, “The 
Design and Use of Special Purpose Processors for the Machine Process¬ 
ing of Remotely Sensed Data,” Proc. Symposium on Machine Processing 
of Remotely Sensed Data, Purdue University, IEEE Cat. No. 73CHO 
834-2 GE, 1973, pp. la-25 to la-42. 





Programming hardware for remote sensing image analysis 

by DAVID G. GOODENOUGH 

Canada Centre for Remote Sensing 
Ottawa, Ontario, Canada 


INTRODUCTION 

The costs of processing large numbers of images can be 
reduced substantially if use is made of minicomputers com¬ 
bined with special purpose hardware and array processors. 
The experience with systems of the Canada Centre for Re¬ 
mote Sensing (CCRS) is drawn upon to show how parallel 
processing of images can be achieved. Data formats are 
described which accommodate a wide variety of imaging 
sensors from scanners to radars. We present the software 
architecture for the movement and analysis in parallel of 
large images stored on disks. One such system at CCRS is 
able to perform a 4-feature, LANDS AT maximum likelihood 
classification of seven million pixels into 93 classes in 15 
minutes and a two-dimensional Fourier transform of a 512 
by 512 image in 12.0 seconds. The array processor which 
achieves these rates is discussed at length including its soft¬ 
ware and hardware components. 

CCRS IMAGE SOURCES AND STRUCTURES 

The Canada Centre for Remote Sensing (CCRS) receives 
image data from the LANDSAT-1 and -2 satellites of the 
United States and from sensors on four CCRS aircraft. The 
primary digital image sources for CCRS are listed in Table 
I. In the near future NASA will launch the LANDSAT-C 
and SEAS AT-A satellites for which we will receive data of 
Canada. The sensor in most demand now is the four-channel 
LANDS AT multispectral scanner (MSS). The synthetic ap¬ 
erture radar (SAR) of SEASAT-A may allow effective all- 
weather sensing of ice, ocean, and land features which make 
this also an important source of data. The satellite image 
sources are characterized by wide area coverage, frequent 
coverage (when compared with an aircraft program) and 
high data rates. CCRS pursues projects involving airborne 
MSS and SAR systems. Each image source produces data 
with unique characteristics and errors. For CCRS remote 
sensing systems, the received data are passed through a 
Ground Data Handling Centre (GDHC) which organizes and 
corrects the data for the most serious errors and produces 
a computer compatible tape (CCT) for subsequent remote 
sensing image analysis. By remote sensing image analysis 
we mean the semi-automatic extraction of information from 
a data set, which may be composed of data from multiple 


sources, and the generation of a tabular, graphical, or digital 
representations of this information for later use by resource 
managers. The primary output products desired from the 
CCRS Image Analysis System (CIAS) are summarized in 
Table II. With the exception of item (4) in Table II, all other 
products can be produced at CCRS by a system involving 
a general purpose minicomputer and two special purpose 
hardware processors. The objective of this paper is to pres¬ 
ent for your consideration observations and conclusions re¬ 
lated to the programming and analysis of remotely sensed 
data in systems with hardware processors. 

The LANDSAT-1 and -2 multispectral scanners 1 gener¬ 
ates images which are framed at CCRS to correspond ap¬ 
proximately to a ground area of 185 km by 181 km. Each 
frame contains approximately 3240 picture samples (pixels) 
along each of 2286 scan lines. The MSS generates through 
24 detectors four spectral bands, the first three (MSS 4, 5, 
6) of which are logarithmically scaled and the fourth band 
(MSS 7) is linearly scaled. The raw spectral values are each 
quantized to 6 bits. The CCRS-GDHC produces correspond¬ 
ing CCTs containing radiometrically corrected, 8-bit quan¬ 
tized images. These LANDSAT CCTs follow a flexible, self- 
defining format 2 often referred to as the CCRS-JSC format. 
The basic form of this format for LANDSAT MSS CCTs is 
summarized in Table III. This CCT format is used by CCRS 
for all digital images and by other countries for LANDSAT 
image production. 

A digitized image or picture of L lines, B bands, and P 
pixels is an integer array which can be represented by the 
real function, G, where, 

G(L,B,P)= £ £ £ f(l,b,p) (1) 

«=1 6=1 p=1 

and f(l,b,p ) is the intensity in band b for a pixel at position 
p on scan line 1. The order of summation in equation (1) 
indicates that the image is primarily interleaved by lines and 
secondarily interleaved by band. Such an image is usually 
said to be line interleaved. In remote sensing image proc¬ 
essing, two other image forms are common, G(L,P,B) with 
secondary interleaving by pixel (pixel interleaved), and 
G(B,L,P) with primary interleaving by band (band inter¬ 
leaved). Different sensors produce different data structures 
which must often be reformatted into the structure most 
compatible with the given image processing system. Such 
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TABLE I.—Primary CCRS Digital Image Sources 
All data are stored on 2400 ft., 1600 bpi computer compatible tapes. 

(1) LANDSAT-1 and -2 Multispectral Scanners (MSS) 

—4 registered spectral bands 

—3240 pixels by 2286 lines = 1 frame 
—8 bits/pixel 

(2) Airborne Daedalus MSS 

—11 registered spectral bands 
—728 pixels by 4500 lines 
—8 bits/pixel 

(3) ERIM Synthetic Aperture Radar 

—4 unregistered bands, 2 frequencies, 2 polarizations 
—4000 pixels by 1800 lines 
—8 bits/pixel 

(4) Aerial Photographs (Color or Black and White) 

—scanned on a microdensitometer 

—3 registered spectral bands 
—4000 pixels by 2400 lines for 3 bands 
—12000 pixels by 2400 Ones for 1 band 
—8 bits/pixel or 10 bits/pixel. 


reformatting can be quite expensive because of the large 
data volumes involved. 

An image contains spatial information, in addition to its 
spectral information, which should also be used in the sub¬ 
sequent data analysis. The balance in information content 
carried by the spatial properties of an image and its spectral 
properties varies from sensor to sensor, image to image, and 
application to application. For each combination of sensor 
and application, it is necessary to study this balance between 
spatial and spectral information. Spatial properties of a dig¬ 
ital image are often characterized by computed spatial fea¬ 
tures which measure the differences or similarities between 
a pixel and its neighborhood. Such spatial features may be 
used as texture measures to be added to the dimensionality 
of the feature set describing the image or they may be used 
to generate edges. Edges can in turn be used to indicate 
homogeneous regions for which averaging of spectral fea¬ 
tures would be appropriate as a data compression aid. 


TABLE II.—Primary Cl AS Output Products 

(1) Computer compatible tapes 

—flexible, self-defining CCRS-JSC format 
—up to 64 bands or features 
—for input to: 

(a) PDS color-write microdensitometer (high resolution) 

(b) CCRS Electron Beam Image Recorder (EBIR) System (medium 
resolution) 

(c) Color Strip Film Recorder (low resolution) 

(d) Grid Structured Geographic Data Bases 

(2) Approximate maps of class distributions 

(a) —scaled for UTM maps from 1:50,000 to 1:1,000,000 
—single class or multiple class representations 

—latitude/longitude reference 

(b) —photographs of CRT display. 

(3) Tables 

(a) measurements of class areas 

(b) summaries of spectral and spatial statistics by class. 

(4) Polygon Data Files 

—generalizations of grid structured classifications for input into large, 
polygon structured data bases 
—transferred by a unique tape format. 


TABLE III.—CCRS-JSC Format for LANDSAT-2 MSS Images 

There are five types of records in the basic 2400 ft., 1600 bpi CCT. 

(1) JSC Header Record 

—3060 bytes long 

—contains information about the subsequent organization of the data 
on the CCT, e.g. the quantization, the processing used, the number 
of bands or features (up to 64), the number of pixels and lines, etc. 

(2) LANDS AT Header Record 

—1440 bytes long 

—gives information specific to CCRS-GDHC processing such as orbit 
number, frame number, image date, special processing flags, frame 
center latitude and longitude, etc. 

(3) Geometric Transformation Record 

—2700 bytes long 

—contains the geometric transformation parameters necessary to move 
from LANDSAT coordinates to the Universal Transverse Mercator 
(UTM) grid. 

(4) Radiometric Look-up Table Records 

—there are 4 radiometric look-up table records, one for each spectral 
band 

—each record is 1620 bytes long 

—in each record the radiometric look-up tables for the six detectors for 
that spectral band used to correct this CCT are given. 

(5) MSS Data Sets 

—a data set is made up of 4 records, each 3780 bytes long 

—each record contains the record number, ancillary data, and video 
data for each spectral band 

—the ancillary data includes GMT start scan time, latitude and longi¬ 
tude at the line center, satellite altitude and roll, pitch, yaw, and 
calibration wedge data. 


Adding dimensionality to the data set through the addition 
of texture features increases the processing costs not only 
by increasing data volume into the analysis system, but also 
by increasing the costs of ground truth acquisition. The 
higher dimensionality of the feature set requires that more 
training samples be collected to maintain the same level of 
accuracy in the elements of the covariance matrix in order 
to achieve higher classification accuracies with additional 
features. 3 More training samples require the detailed ex¬ 
amination of larger areas on the ground. In addition, in order 
to have confidence in the statistical estimates of classifica¬ 
tion error, test sites developed from ground truth should be 
increased in corresponding size to the increase in size of the 
training sites. Error estimates of classifications from training 
sites are overly optimistic. Similar error of classification 
estimates from test sites are realistic if the procedures de¬ 
scribed in a previous paper 4 are followed. 

The image tape is transferred to disk in a CCRS file struc¬ 
ture family known as UNIDSK. 5 This is a flexible, grid data 
base structure which can accommodate up to 64 features. 
Any form of interleaving is supported by this structure. The 
file contains a Master Header, Secondary Headers, a Video 
Header, and the Video Data Set. The Master Header, 1024 
bytes long, identifies the file, its format, and the geographic 
location. The Secondary Headers are variable in number 
and length and contain information specific to the history of 
this file and pointers to related files. The Video Header 
contains the details of the format and preprocessing of the 
video data in the following Video Data Set. This family of 
formats also allows random access of pixel values, carries 
sufficient information to allow-one to re-trace the steps by 
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which the file was created, and permits images to continue 
onto multiple disk packs. This UNIDSK structure is used 
for storage and analysis of all images at CCRS. 


CIAS OVERVIEW 

The CCRS Image Analysis System (CIAS) 6 consists of 
two minicomputer sub-systems, the IMAGE 100 sub-system 
and the PDS sub-system. The PDS sub-system includes a 
DEC PDP-11/40 minicomputer and a PDS flatbed color read/ 
write precision microdensitometer. The PDS sub-system is 
connected to the IMAGE 100 sub-system via two paths, a 
dual port disk drive and a high speed (1 Mbit/sec) interpro¬ 
cessor communication link (DMC 11). The PDS sub-system 
is used either to digitize color aerial photographs and pro¬ 
duce a UNIDSK file for subsequent analysis on the IMAGE 
100 sub-system or to write such a file onto color film. 


The IMAGE 100 sub-system, shown in Figure 1, is used 
for all image analysis. This sub-system is controlled by a 
DEC PDP-11/70 minicomputer with 256K 16-bit words of 
core memory. There are two additional processors in this 
system, a modified General Electric IMAGE 100 and a 
CCRS array processor called the Image Analysis Processor 
(IAP). There are three parallel paths for data transfers, the 
UNIBUS, the RH70/DWR70 data bus, and the IAP-RH11- 
DB1 data path. The IMAGE 100 is an interactive pipeline 
processing system 6,7 which supports the acquisition and non- 
parametric classification of 512 bv 512 four-feature images. 
The pipeline processor permits real-time ratioing of features 
and 4 by 4 matrix multiplication to generate linear combi¬ 
nations of features. 

Classification of the image is performed by comparing in 
hardware all image pixels with the signature distribution of 
the training area as represented by the distribution of 4- 
vectors or cells of unit volume. The frequencies of occur¬ 
rences of 4-vectors may be thresholded to alter the resulting 


CIAS IMAGE 100 SUB-SYSTEM 



The other two processors in the sub-system are a modified General Electric IMAGE 100 and a CCRS/CDC array processor, the Image Analysis Processor (IAP). 
Note the parallel data paths permitting simultaneous image operations. The large disk, DB1, is dual ported to the IAP and serves as one of two image storage 
locations. As a result, the IAP can perform a maximum likelihood 93-class classification of LANDSAT four-feature frame in 15 minutes. Connections labelled 

PDS indicate links to the PDS sub-system of the CIAS. 
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classification. The resulting classification may be stored as 
a 1-bit overlay. The IMAGE 100 is very interactive and is 
particularly rapid for signature extraction. In supervised 
classification, a hierarchical approach is followed at CCRS 
from simple rectangular parallelepiped classification to non- 
parametric cell classification to maximum likelihood classi¬ 
fication. The maximum likelihood classification may be per¬ 
formed in software in the PDP-11/70 or in hardware in the 
IAP. Unsupervised classification or clustering is performed 
in the following fashion. An image area is selected and the 
vector distribution determined for this area. Clustering by 
non-parametric or parametric methods of this distribution is 
carried out in the 11/70. The resulting class statistics can be 
loaded into the IAP and the classification executed on the 
whole image stored on the large disk labelled DB1 in Figure 
1. Alternatively, the non-parametric class boundaries can be 
loaded into the IMAGE 100 and the 512 by 512 scene is then 
classified. Prior to the development of the IAP most of the 
system time was devoted to the maximum likelihood clas¬ 
sification in the PDP-11/70 of large images. 

The multiple processing capabilities of the CIAS are fully 
utilized under the DEC RSX-11D Version 6.2 multitasking 
operating system. The IMAGE 100 is controlled through an 
executive task. The software is written in DEC FORTRAN 
IV PLUS with the exception of a few input/output routines 
in MACRO-11. FORTRAN IV PLUS was selected because 
of the ease of transportability of such programs within CCRS 
and with our many user agencies and because of the efficient 
code generated by the FORTRAN IV PLUS compiler. For 
example, maximum likelihood classification executed three 
times as fast on an 11/70 in FORTRAN IV PLUS compared 
with DEC FORTRAN IV (FOR) on the same computer. The 
standardization of tape and disk formats for all our com¬ 
puters meant that software developed on our PDP-10/KI 
could be brought up in a day on our PDP-11/70. The CIAS 
interacts with the operator through program selection from 
a menu of programs presented on a Tektronix 4012 graphics 
terminal or by depressing any combination of function 
switches on the IMAGE 100. The menus available in the 
IMAGE 100 sub-system are listed in Table IV. Each menu 
is selected by letter. Each task listed in the menu is selected 
by number. A task in a menu may itself call a group of tasks. 
This approach can be contrasted with the use of complex, 
independent command driven tasks or picture processing 


TABLE IV.—CIAS Image 100 Sub-System 
Menus 

A. Initialization/status 

B. Input/output 

C. Preprocessing 

D. Spatial Feature Generation 

E. Feature selection 

F. Signature acquisition 

G. Classification and clustering 

H. Classification filtering 

I. Utility 

J. Diagnostic 

K. Application menu # 1 

L. Application menu #2 


languages. The menu approach is simple to implement, easy 
to maintain, and easily expanded. The operator is not re¬ 
quired to learn a new language or symbol set, yet the tasks 
are grouped in a logical fashion to guide the operator in their 
use. In addition, the operator can always select directly 
tasks by name or by a menu letter-number combination. The 
two Application Menus are collections of tasks to support 
a particular remote sensing application such as forest inven¬ 
torying, crop acreage estimation, and so forth. 6 

Image input to the IMAGE 100 sub-system is via the two 
125 ips tape drives or the three disk drives. Training area 
selection, ground control points, and other map information 
may be input directly from the large digitizer tablet. Output 
maps and plots are generated on one of the two printer/ 
plotters. Statistical summaries are produced on the line 
printer. The disk configuration permits parallel processing 
to be achieved and is discussed in the next section. 

CIAS DISK CONFIGURATION 

The arrangement of the three moving head disks in the 
CIAS is shown in Figure 2. Two of the large RP04 disks (88 
megabytes) are dual ported. The disk unit DB0 is the system 
disk and has only one port. Parallel processing is achieved 
by connecting the Image Analysis Processor (IAP) to DB1 
through the UNIBUS-B port of the RH11 disk controller. 
The PDP-11/70 through execution of programmed instruc¬ 
tions directly controls all interfaces and thus, indirectly, all 
data transfers within the system. The core memory contains 
the programs under execution and the data buffers to be 
used by the Direct Memory Access (DMA) controllers, such 
as the RH70, RH11, and DR11B. Most data transfers be¬ 
tween two devices in the CIAS use core memory as a tran¬ 
sitory repository. Memory speed is thus a limiting factor on 
system data throughput with one exception, the IAP/DB1 
link since it bypasses the core memory completely. 

In Figure 2, the letters “D” and “C” are used to indicate 
paths along which data or control information, respectively, 
can flow. Control information between the PDP-11/70 and 
the three DMA controllers moves along UNIBUS-A. Data 
transfers from the DR11B and RH11 controllers to the 11/70 
can also be carried out along UNIBUS-A. The impression 
of parallel transfers is created by the interleaving of these 
data transfers along UNIBUS-A to and from main memory. 
Eventually such transfers saturate the UNIBUS-A at a data 
rate of approximately 600 K words/sec. 8 The RH70 disk 
controller sends and receives control information along UN¬ 
IBUS-A and Massbus-A, but directs data transfers along the 
Fastbus and Massbus-A. The RH11 disk controller com¬ 
municates control information along UNIBUS-A but it may 
be selectively directed under program control to accept data 
transfers along either UNIBUS-A, UNIBUS-B, or Massbus- 
B. The two disk controllers and two data buses make it 
possible to have simultaneous data transfers to and from 
two disks. 

Since the manufacturer of the dual port disks did not 
provide supporting dual port software, it was necessary for 
us to develop our own. The dual port disk DB1 can be either 
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Figure 2—The details of the 88 megabyte disk connections for data (D) and control (C) information transfers are shown. The IAP can access DB1 while the 11/ 
70 is simultaneously accessing DBO or DB2. The IAP is also connected directly to the IMAGE 100. The RH11 and RH70 are DMA disk controllers. One disk 
handler handles all disk requests. Images are stored on disks in UNIDSK format (see text) compatible with DEC Files-11 structure. DB2 is used primarily for 
image transfers between the two sub-systems. Communication information concerning DB2 is passed along the interprocessor link, the DMC11, between two 

CPUs. 


attached to the RH70 controller or the RH11 controller or 
unattached. In the unattached state either controller can 
attach the drive, at which point the other controller cannot 
access the drive. The dual port disk DB2 is interconnected 
between the PDP-11/70 and the 11/40 computers. This disk 
is used primarily for image transfers between the two sub¬ 
systems and can, therefore, be dedicated to either computer 
for long periods of time. Communication information con¬ 
cerning DB2 is passed along the interprocessor link, the 
DMC11, between the two CPUs. For all images on disk, the 
UNIDSK format is used. This format has been implemented 
to generate files compatible with the DEC standard; namely, 
Files-11. The DEC disk handler, DB, passes file control 
functions to its ancillary processor task (F11ACP). Both DB 
and F11ACP have been modified to allow two additional 
functions which cause data transfer between the IAP and 


DB1. Dual port conflicts at DB1 are presented by forcing 
F11ACP to schedule sequentially all requests from a user 
task for DB1 data transfers. The IAP handler controls the 
DR11-B and RH11 controllers. The IAP handler does not 
directly accept from a user task requests for DB1 data trans¬ 
fers but receives such requests via F11ACP. The software 
development described here has enabled us to achieve the 
hardware data transfer rate limit of the RP04 of 2.5 /usee/ 
word. The IAP is described in detail in the following section. 

CIAS IMAGE ANALYSIS PROCESSOR (IAP) 

CCRS, in cooperation with Computing Devices Company, 
have designed and implemented a special purpose array 
processor, the IAP. The IAP was intended to meet three 
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processing needs: (1) classification with the maximum like¬ 
lihood decision rule; (2) computation of two-dimensional 
Fourier transforms; and (3) selection of pixels from disk 
images and the transferring of them to and from the IMAGE 
100 solid state memory. An important cost consideration in 
the design of the IAP was to determine the number of bits 
to be carried in the arithmetic unit for the computations. 
The effects of coefficient quantization and arithmetic round 
off for the Fast Fourier Transform algorithm have been 
discussed previously. 10 The IAP was to calculate the FFT 
with decimation in time. The input data for the FFT were 
to be rearranged within the IAP to be in bit-reversed order 
prior to the pipeline arithmetic unit. The IAP would need to 
compute up to a two-dimensional forward and inverse FFT 
of a 512 by 512 input image and produce a result that agreed 
within 1 percent with that obtainable with 32-bit floating 
point arithmetic. Theoretical calculations for the FFT sug¬ 
gested that 18-bit floating point would likely be required. 
The IAP pipeline processor was then simulated on a PDP- 
11/40 for the FFT and the maximum likelihood calculations 
for four-feature images with 8-bit quantization. 

The logarithm of the likelihood of observing a vector x 
given x is from class i with mean covariance matrix C j5 
and a priori probability P if is given by: 10 

l(xli)=^ In 27r-£ln | C 4 | +ln P t 
-£(x- fhiYCf^ix- rhi ) 

The maximum likelihood decision rule is a comparison of 
this likelihood in which the vector x is assigned to the class 
for which the likelihood is a maximum and exceeds a se¬ 
lected threshold. The accuracy and speed of the computation 
is concerned chiefly with the last term, so that the preceding 
equation can be cast in the form for 4-feature data as: 

2 (y t Q«yt)+K 

»=i j=i 

where K is a constant, Q tj is the ijth element of the inverse 
covariance matrix, and y n = x n -m n . Because of the 16-bit 
minicomputer and the ease of 16-bit data manipulation, 16- 
bit fixed point with scaling for this equation was stimulated 
for the IAP pipeline processor with the number of bits for 
the G-matrix and y values varied from 6 to 14 and 6 to 11 
bits, respectively. The product was always truncated to a 
16-bit integer. It was found that 16-bit fixed point with scal¬ 
ing was unsatisfactory and always produced results on real 
data for which the likelihood value differed by more than 10 
percent from those obtained by a 32-bit floating point. With 
18-bit floating point, the likelihood value was always within 
0.1 percent of that obtained with a 32-bit floating point. 

A simulation of the IAP pipeline processor for the FFT 
algorithm was then performed. The two-dimensional dis¬ 
crete Fourier transform, X(k,l ) is given by: 11 

M—l N-l 

X(k,l)= I I x{m,n)W M km W N ln 

m= 0 n=0 

with k= 0, l, , M- 1 and /=0, 1, . . . , N-l and where 


x(m,n ) is the periodic input sequence, and W N =exp(-j2ir/ 
N). The two-dimensional FFT (2-D FFT) was simulated by 
performing two forward FFTs followed by two inverse FFTs 
on sample 512 lines of 8-bit input data. Figure 3 is an ex¬ 
ample of the results obtained by this simulation. The ordi¬ 
nate axis is the root-mean-square variation of the absolute 
error in percent for the real image (RSMR). The absolute 
error in feature intensity in percent for the real image 
(APER) 8 when compared with the real image intensity re¬ 
sulting after a forward and inverse 2-D FFT was applied to 
the real input image is defined by: 


APER= 100 


Real Input-Real Result 
Real Input 


The maximum value of RSMR on the ordinate axis is 0.226 
percent and the minimum value is 0.008 percent for 512 
input 8-bit samples. The solid and dashed line curves cor¬ 
respond to having 5 bits or 6 bits for the exponents(E) of 
the floating point numbers, respectively, with a variable 
number of bits in the mantissa (M). It is clear from this 
figure that the error for E=5, M=13 is lower than that for 


NORMALIZATION TYPE 1 
RMSR 

MAX = 0.226 MIN = 0.008 # 0E SAMPLES = 512 



Figure 3—The IAP arithmetic pipeline was simulated for two-dimensional 
Fast Fourier Transforms (2D-FFT) by performing two forward FFTs followed 
by two inverse FFTs on sample 512 lines of 8-bit input data. The objective 
was to determine the number of bits required in the IAP computations to 
obtain the same result, to within 1 percent, as that obtained on a 32-bit 
machine. This figure is an example of the results obtained. The ordinate axis 
is the root-mean-square variation of the absolute error (RSMR) in percent for 
the real image after a 2D-FFT. The maximum value of RSMR here is 0.226 
percent and the minimum value is 0.008 percent. For 18-bit floating point the 
real and imaginary image errors were always less than 1 percent. From this 
figure we see that 5 bits in the exponent and 13 bits in the mantissa gives the 
lowest error for an 18-bit floating point calculation. 
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E=6, M= 12. A similar result was obtained for the imaginary 
images. We, therefore, selected 18-bit floating point with 5 
bits in the exponent and 13 bits in the mantissa. 

The interfaces of the IAP are indicated in Figure 4. The 
lettered lines connect to those in Figure 5 in which the IAP 
control computer and pipeline processor are shown. The 
IAP is interfaced to the PDP-11/70 for control and data 
through the DR11B. The IAP is interfaced to the RHll so 
that the RP04 disk may be used for storage and retrieval of 
images in parallel with other CIAS activity. The IAP is also 
interfaced to the IMAGE 100 processor through the Line 
Buffer Memory (LBM). The LBM is a high speed random 
access memory (RAM) which can store 40% bytes of image 
data at the 10 MHz data rate required to buffer the Intel 10 
Megabit recirculating memory of the IMAGE 100 processor. 
The IAP can function either independently of the IMAGE 
100 processor or in combination with it to use the ratioing 
and matrix multiplication hardware. The FIFO or First-In- 
First-Out memory receives data and control information 
from the DR11-B or data from the RHll. The FIFO is a 
1000 18-bit word RAM which buffers the flow of data to 
permit the IAP’s control computer to service higher priority 
activities. 

The IAP Control Computer (CC) and Pipeline Processor 
(PLP) are shown in Figure 5. The programmable CC controls 
the initialization of registers which in turn control the other 
elements of the IAP. The CC performs any format conver¬ 
sion required of the data and is used to transfer data from 
one memory to another. 8 The instruction word for the IAP 
is 18-bits long. A cross-assembler between DEC Macro-11 
style instructions and the IAP’s microcode is used to sim¬ 
plify the programming of the CC. The Program Memory 
(PGM) shown in Figure 5 is used primarily to store the 
programs of the CC. The PGM consists 8 of 256 words of 
read-only-memory to store interrupt and bootstrap programs 
and 1 K 18-bit words of RAM. 

The Pipeline Processor is an 18-bit floating point arith¬ 
metic unit composed of four major sections, the Pipeline 
Memory System (PLMS), the Scratch Pad Memory System 
(SPMS), the Pipeline (PL) and the Output Buffer. Once the 
PLP has been initialized by the CC, the PLP can perform 
independent computations. Thus, the CC and PLP can be 
operating in parallel. The PLP has a throughput rate of 100 
ns which is achieved by having each section operate in 
parallel. The PLMS includes three 18-bit 1 K RAMs, marked 
as PLM1 to PLM3 in Figure 5. For maximum likelihood 4- 
feature, 8-class calculations, for example, PLM1 would con¬ 
tain 256 4-feature pixels, PLM2 would store 8 4-feature 
means, and PLM3 would have 8 inverse covariance matrices 
plus the decision rule threshold. Complex address calcula¬ 
tors in the PLMS ensure that the address sequence required 
for FFTs is performed correctly. Data from the PLMS is 
called for by the SPMS which then delivers the data to the 
Pipeline according to the sequence of instructions held in 
the Scratch Pad (SP) memories. The SPs can hold up to 32 
16-bit instructions plus 8 18-bit data words. Each SP pro¬ 
gram instruction is executed in 100 ns so that data may be 
passed to the Pipeline at that rate. The data from SP3 in 
Figure 5 is delayed in order to arrive at the second multiplier 


coincident with the output from the first multiplier. The 
output of the second multiplier is a 23-bit floating point 
number with 5 bits for the exponent and 18 bits for the 
mantissa. To achieve the required speed in the accumulator, 
it was found necessary to convert from a floating point 
number to a fixed point (47 bits) number (unfloat in Figure 
5), accumulate, and then convert back to an 18-bit floating 
point number. When the required number of terms have 
been summed, the computed likelihood value, for example, 
is passed to the Output Buffer. The Output Buffer simplifies 
the programming of the PLP and determines the maximum 
likelihood value, thresholds this value, and outputs the re¬ 
sulting class number (16-bit integer) and likelihood to value 
to PLM1. The class number may be coded as an integer or 
with one bit set corresponding to the class. In the FFT mode 
the real and imaginary values are returned to PLM1 and 
PLM2, respectively. Thus, as data are flowing out of the 
PLMS, results are flowing into the PLMS. When the data 
have been processed, the array of results are sent to DB1, 
the IMAGE 100 or to the PDP-11/70 memory. 

The size of memories in the PLP restrict for maximum 
likelihood the number of features and classes which may be 
accommodated in one pass. The number of terms, N T , in 
the maximum likelihood expression for N features is: 
N t = : N(N+ l)/2. If we assume one SP instruction is required 
for each term, then only 32 terms can be accommodated by 
the SP memory. If N=1 features, then N T =28 terms which 
means the IAP is limited to 7 features in one maximum 
likelihood pass. The maximum number of classes, M max , for 
one pass is limited by N and by the size of the PLM ac¬ 
cording to: M max -(Ar r +l)< 1024. For four features, 
Mnax = 93 classes. Other examples are listed in Table V. The 
PLP can calculate the maximum likelihood function for M 
classes and N features at a rate of 10 _7 -M-(/V r +l) seconds 
per pixel. Program and input/output overheads increase this 
time for large images as is evident in Table V. 

The IAP, although originally designed for rapid quadratic 
and FFT calculations, has other important features. The IAP 
can perform table look-up in 200 ns through one CC instruc¬ 
tion. This is useful for radiometric and atmospheric correc¬ 
tions, image enhancements, and other image intensity ma¬ 
nipulations. The IAP can reformat data simultaneously as 
they are moved from one memory to another which reduces 
substantially the processing costs associated with different 
input formats. The IMAGE 100 can display up to 8 classes 
as colored 1-bit overlays over a 512 by 512 pixel scene. 
The IAP can convert rapidly class numbers into 1-bit overlay 
patterns for subsequent IMAGE 100 display. 

In classifying an image, the CIAS can first be used in a 
learn mode with sub-images displayed on the IMAGE 100. 
In this mode representative areas for supervised classifica¬ 
tion or for clustering are selected and parametric statistics 
for each class or cluster generated. The supervised classi¬ 
fication is interactive and flexible. Parameters entered in 
response to teletype requests for a program are entered in 
files for subsequent utilization by that program and the IAP 
in a bulk processing mode. 8 For the classification of an N- 
feature, 512 by 512 scene, the statistics and input video files 
are read by the IAP and the scene classified. Figure 6 shows 




Figure A —The IAP is interfaced through the First-In-First-Out (FIFO) 1 K word memory to the PDP-11/70 through the DR11B and to a large disk through the 
RH11 disk controller. The IAP is also interfaced to the IMAGE 100 through the 4 K bytes of Line Buffer Memory (LBM). This latter connection allows the IAP 
to use the ratioing and matrix multiplication hardware of the IMAGE 100, to use the 10 megabit Intel memory, and permits rapid image transfers between the 

IMAGE 100 and the disk DB1 through the IAP. The lettered lines connect to Figure 5. 
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CONTROL COMPUTER PIPELINE PROCESSOR 

Figure 5—The IAP’s Control Computer (CC) and Pipeline Processor (PLP) are shown. The CC is programmable and controls the initialization of registers which 
in turn control the other elements of the IAP. PLMs are 18-bit 1 K RAMs which hold the data, means, and covariances for maximum likelihood, for example, 
and in which the results from the output buffer are stored. Scratch Pad (SP) memories hold instructions on which PLM to select and deliver data to the pipeline. 
The pipeline computes quadratic summations at 100 ns per term. The IAP can perform a forward 2D-FFT of a 512 by 512 8-bit input image in 12 seconds. It can 
also perform a table look-up in 200 ns and reformat data rapidly as they are moved from one memory to another. 
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TABLE V.—CIAS/IAP Processing Times 


I. Maximum Likelihood Classification: 
(excludes statistics generation) 

No. of 
Classes 

No. of 
Channels 

Line 
Length' 

No. of 
Lines 

Processing 

Times 

LANDS AT-1,2 

32 

4 

512 

512 

12 sec. 


32 

4 

3300 

2286 

5.8 min. 


93(max) 

4 

3300 

2286 

15.0 min. 

LANDSAT-3 

64(max) 

5 

3300 

2286 

15.2 min. 

MSS(1 Pass) 

35(max) 

7 

2048 

2048 

8.2 min. 

Multisensor or Multitemporal 
(3 Pass) 

32 

11 

2048 

2048 

20 min. 

II. 2-D-FFT 


1 

512 

512 

12.8 sec. forward 
17.0 sec. inverse 

III. IMAGE 100 Transfers from Disk 

Decimate a LANDSAT Frame to: 


4 

512 

512 

28.0 sec. 

Move an I-100 Screen to: 


4 

512 

512 

12.0 sec. 


how the processing times for the maximum likelihood de¬ 
cision rule (MLDR) vary with number of features (4 to 7) 
and number of classes. The stars in Figure 6 mark the max¬ 
imum number of classes which can be processed in one pass 
for the number of features indicated. At low numbers of 
classes one is limited by the minimum time (2.5 seconds) 
required to input the image to the IAP. The classification 
results flow to core memory and then to disk. It is of interest 
to note that for more than 21 classes, an image on a 1600 
bpi magnetic tape can be directly read in on our 125 ips 
drives and classified by MLDR with no loss in processing 
speed. 

Table V summarizes the processing rates achieved by the 
CIAS with the IAP for maximum likelihood classification, 
2-D Fast Fourier Transforms, and transfers from the large 
DB1 disk to the IMAGE 100 solid state memory. These 
rates are high enough that most of the CCRS bulk processing 


MLDR CLASSIFICATION 
RESULTS TO CORE MEMORY 



TIME SECONDS- 

Figure 6—The variation of IAP processing times are shown for maximum 
likelihood decision rule (MLDR) classification for normal distributions for a 
512 by 512 image of N features with 8 bits per feature. The number of features 
has been changed from 4 to 7. The stars mark the maximum number of 
classes the IAP can process for the indicated feature space dimensionality. 
At low numbers of classes one is limited by the minimum time (2.5 seconds) 
required to input the image to the IAP. The classification results flow to core 
memory and then to disk. 


requirements for LANDSAT data can now be met by the 
CIAS. The parallel processing permitted in the CIAS by the 
11/70, IMAGE 100, and IAP and the parallel processing 
within the IAP itself mean that the rates (within 10 percent) 
given in Table V are achieved even under full system load. 


CONCLUSION 

The minicomputer when combined with an array processor 
and display hardware can match or exceed the image proc¬ 
essing performance of larger, general purpose computers. 
The complexities caused by the additional hardware can be 
reduced by following programming standards and conven¬ 
tions consistent with the software operating environment 
selected for the system. The use of a single standard tape 
format and a single disk format minimizes the number of 
input/output programs required to support the additional 
hardware. The inclusion of parallel processing and multi¬ 
tasking permits a minicomputer based system to achieve 
phenomenal image processing throughput at lower costs 
than conventional systems. 

Flexibility can be maintained without inordinate sacrifices 
in speed. It is often expressed that software image process¬ 
ing systems are much easier to change than systems with 
special processors. Our experience has been that software 
systems of hundreds of programs developed a rigidity com¬ 
parable to systems with hardware processors. Both ap¬ 
proaches require that good documentation and change pro¬ 
cedures be followed. 

The application focus of this paper has been image proc¬ 
essing in remote sensing. Many of the techniques presented 
here are, however, applicable to other areas of image proc¬ 
essing and can be used in those areas to obtain lower cost 
solutions to their image processing problems. 
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Use of textural features in the analysis of Landsat images 


by H. K. RAMAPRIYAN, S. H. CHANG, and R. L. McKINNEY 

Computer Sciences Corporation 
Falls Church, Virginia 


INTRODUCTION 

The use of remotely sensed data for land-cover categoriza¬ 
tion has been well demonstrated in recent years. 1 In addition 
to the use of conventional methods, such as windshield 
surveys and low- and high-altitude aerial photographs, sat¬ 
ellite data are being employed by several planning agencies 
for creating land-use and land-cover data bases to assist in 
decisionmaking. 

Human interpretation of aircraft and satellite imagery is 
still used extensively and is essential for certain applications 
(e.g., lineament studies in geology). However, automatic 
interpretation of the image data using digital computers sig¬ 
nificantly complements human analysis and is advantageous 
in providing consistency, labor savings, and timely availa¬ 
bility of interpreted data for postprocessing and inclusion 
into data bases. 

The major differences between human and machine inter¬ 
pretation of images are as follows. A human interpreter uses 
color, spatial variation of color, and relative location of 
objects (i.e., spectral, textural, and contextual features) in 
an image simultaneously. Starting with a synoptic view of 
the image, the human interpreter partitions that image into 
homogeneous segments and associates labels with each of 
them. It is difficult to design algorithms for machine inter¬ 
pretation which simulate the human photointerpreter. How¬ 
ever, machine analysis has the advantage of high resolution 
not achievable with human interpretation. Although it is 
possible to incorporate spectral, textural, and contextual 
features into machine analysis, the most commonly used 
classification algorithms employ only the spectral data and 
assign categories (or class labels) to single resolution ele¬ 
ments (pixels) independently of other pixels. 

The accuracy of classification maps is a principal concern 
for users. Studies of classification algorithms have been 
conducted which compare ground-truth maps with the clas¬ 
sification maps generated by machine interpretation of 
Landsat multispectral data using only the spectral features 
at individual pixel locations. For example, in Reference 2, 
confusion is shown to exist between certain urban and ag¬ 
ricultural cover types. This is due partially to the similarity 
in the spectral characteristics and partially to the differences 
in human and computer characterization of classes (e.g., 
when a region with large residential lots is encountered, 
human interpretation for the entire region is “urban,” 


whereas machine interpretation is dependent on the urban- 
vegetation mixture). To distinguish between categories ex¬ 
hibiting similar spectral reflectances, the use of additional 
features dependent on texture and/or context is helpful. For 
example, although the spectral characteristics of some urban 
and agricultural cover types are similar (due to the variety 
of objects constituting urban pixels and the relatively ho¬ 
mogeneous nature of agricultural regions), the variations of 
spectral reflectances within an urban region can be expected 
to be greater. 

In the past several experiments have been performed on 
the use of textural features in classification. A review of the 
work on textural feature definition is given in Reference 2. 
In Reference 3, a comparative study of texture measures 
used for terrain classification is presented; the features are 
categorized as based on Fourier power spectra, second- 
order grey-level statistics, grey-level differences statistics, 
and grey-level run-length statistics. Examples of textural 
features based on Fourier transforms of image segments are 
given in References 3-5. In Reference 4 textural features 
invariant under translation and rotation are derived in terms 
of discrete Fourier transforms of square image segments and 
are shown to improve the accuracy of classification. Several 
studies by Haralick and his coinvestigators 6-12 employ var¬ 
ious textural features defined in terms of grey-level co-oc¬ 
currence matrices which can be regarded as second-order 
grey-level statistics; “contrast,” “angular second moment,” 
“entropy,” and “correlation,” which are functions of such 
matrices, are demonstrated as particularly useful. Grey-level 
difference statistics, defined in Reference 3, are obtained in 
terms of the frequencies of occurrences of particular values 
of grey-level differences at given separations and orienta¬ 
tions in an image segment. The features derived from the 
grey-level co-occurrence matrices and the grey-level differ¬ 
ence statistics are somewhat similar. Other measures of 
texture are defined in Reference 13 in terms of the frequen¬ 
cies of “runs” of grey-levels in given ranges in specified 
directions. 

The utility of these textural features has been demon¬ 
strated for terrain classifications using aerial photography 
and Landsat data. However, in all of these studies, the 
features have been defined over large blocks of data, which 
implies a loss of effective resolution. The purpose of this 
paper is to develop a set of features which are a compromise 
between using large blocks of data (reducing resolution) and 
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using only spectral information at a single resolution cell. 
These features will be defined at every point in the image 
but will use combinations of spectral reflectances at neigh¬ 
boring pixels of each pixel. Thus, an augmentation of the 
conventional spectral features with spatial and, in turn, tex¬ 
tural information results. Preliminary experimental results 
on the reduction in dimensionality of such feature vectors 
and a comparison of classifications generated by the con¬ 
ventional and the proposed techniques are presented in the 
following sections. 


STATEMENT OF THE PROBLEM 

The notation to be used in the remainder of this paper is 
as follows. Let P denote the array of spectral measurements, 
with p m denoting the kth spectral feature measured at the 
ijth pixel, where i equals 1,2 , . . . , M; j equals 1, 2, . . 

. , N\ and k equals 1, 2, . . . , K (in the case of Landsat, K 
equals 4). 

Conventional classification techniques convert P to an 
array, Q, called the classification map, where <? y are num¬ 
bers in the range 1 to L, i and j have the same range as in 
the case of P, and L equals the number of classes. The 
decision on the assignment of p ijk to qy depends only on 
the K numbers corresponding to the i jth pixel in P. 

The problems considered in this paper are (1) how to 
define features at every point in the image such that the 
original resolution of Pis preserved while, at the same time, 
spatial dependences of spectral reflectances are utilized in 
making the class assignments in Q and (2) how to use these 
features to obtain improved classifications without signifi¬ 
cant increases in computation. 


FEATURE DEFINITION 

In terms of the notation introduced previously, simple 
textural augmentation of spectral features can be provided 
as follows. For each ij define a feature vector, Xy, given by 

x ij~{Pij I Pu+l I Pi+lJ I Pi+lJ+ll (1) 

where py represents the ^-dimensional feature vector at ij 
(assuming all vectors to be row vectors). 

Thus, with Landsat data 16-dimensional feature vectors 
result. Other similar feature vectors can be defined at every 
point in the image. Examples are the following: 


x ij {Pij 

1 Pii 

Pi,}+ 1 1 1 Pij Pi+lJ 1 

1 Pa ~ Pi+ u+i |} (2) 

x ij {.Pij 

Pi-lJ 

Pij— 1 Pi+lJ PiJ+ 1 } 

(3) 

x ij = {Pij 

Min 

fc,i=±i 

Pij-Pi+kj+i 1 Max | 

fc,i=±i 

Pij Pi+kJ+1 } (4) 


(In all of the foregoing equations, special definitions must 
be made for the cases in which the subscripts of p exceed 
their ranges. A reasonable definition is to replace a+u by 
Pij and so on.) 

Each of these feature vectors includes the spectral fea¬ 


tures at the point i j as well as those in the immediate sur¬ 
roundings of ij. These vectors could simply be used in a 
conventional classification algorithm such as a maximum- 
likelihood supervised classifier. However, because the di¬ 
mensionality of the feature space has effectively been in¬ 
creased by a factor of four (five and three in the case of 
Equations (3) and (4), respectively), it is desirable to reduce 
the dimensionality by some feature selection or extraction 
technique and then perform the classification. 

In the experiments discussed here, the feature vector def¬ 
inition (Equation (1)) is used. The Karhounen-Loeve (K-L) 
principal components transformation 14 is used to reduce the 
dimensionality. This linear transformation is derived from 
the orthonormal eigenvectors of the covariance matrix, C, 
of the 4K-dimensional data. By arranging the eigenvectors 
such that they correspond to the eigenvalues of C in de¬ 
scending order, the first few components of the transformed 
feature vector can be used for classification, because most 
of the significant information is contained therein. An alter¬ 
native method is a canonical transformation, 15 the use of 
which facilitates separability of the categories of interest. 
This transformation is derived using the statistics of the 
individual classes to be separated. The transformed com¬ 
ponents are such that the separations among classes are in 
descending order from the first to the last component. The 
first few components can therefore be chosen for classifi¬ 
cation. The K-L transformation was used here because the 
software implementing it was readily available. 

EXPERIMENTAL RESULTS 

A 250-by-250-pixel subset of a Landsat scene (ID 1999- 
15091) covering Orlando, Florida, was used in this experi¬ 
ment. The four bands of the image are shown in Figure 1. 
Using the feature definition of Equation (1) and the K-L 
transformation, textural features (the features including the 
spatial and spectral information) were extracted. The first 
eight of the individual components so derived are shown in 
Figure 2. The first two components are effectively low-pass 
filter output, whereas the subsequent components tend to 
emphasize the edges of the image. This is also evidenced by 
an examination of the eigenvectors used for the transfor¬ 
mation as shown in Figure 3. 

Using training samples, eight classes were defined in this 
scene: 


Class 

Name 

Description 

1 

Residential 1 

Residential with significant tree 
cover 

2 

Residential 2 

Residential with sparse tree 
cover 

3 

Vegetation 1 

Dense trees and shrubs 

4 

Vegetation 2 

Dense trees 

5 

Vegetation 3 

Dense shrubs 

6 

Vegetation 4 

Sparse trees and shrubs 

7 

Water 

Water 

8 

Swamp 

Mixture of wetlands and 
vegetation 


The image was classified using the spectral and textural 
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Figure 1—A subset of the Landsat scene covering Orlando, Florida 
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Figure 2—First eight principal components after textural feature 
augmentation 
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Figure 3—Eigenvectors defining the K-L transformation 



Figure 4—Classification map using raw Landsat channels 



features separately. The two resulting classification maps 
(CMs) are shown in Figures 4 and 5, respectively. The major 
differences between the two CMs are in the swamp (class 
8)/vegetation (class 3) area on the left side of the image and 
in the increase of residential classifications in Figure 5 rel¬ 
ative to Figure 4. The main increase in the residential classes 
comes from the locations in Figure 4 which were classified 
as vegetation. Although a ground-truth verification was not 
performed in this experiment, a slight overestimate, rather 
than underestimate, of the residential class is desirable (e.g., 
in census applications in which decisions on locations for 
manual data gathering are made based on the CMs). Also, 
the CM in Figure 5 is more uniform within each class than 
that in Figure 4. Figure 6 shows a confusion matrix between 
the two CMs, with the ith row, yth column element equal 
to the number of occurrences of the rth class in Figure 5 and 
the jth class in Figure 4. Defining the similarity measure 
between the two CMs to be the trace of the confusion matrix 
expressed as a percentage of the total number of pixels, it 
is found that the two CMs are 63.4 percent similar. Because 
the major differences between the CMs are in classes 3 and 
8, the spectral and textural features were further investigated 
by generating plots of the spread of data values in each of 
the eight classes. These plots are shown for the spectral 
features in Figure 7 and for the first four textural features 
in Figure 8. It can be seen that, in the row data bands, the 
overlap between classes is greater than in the case of textural 
features, especially for classes (1, 2) and (3, 8). Because the 
separability of classes is better in the case of textural fea¬ 
tures, it is highly likely that the classifications in Figure 5 
are more accurate than those in Figure 4. 


CONCLUSIONS 

Simple features using spectral and spatial information have 
been defined in this paper. A difference between the textural 
features defined herein and those defined in previous works 
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Figure 5—Classification map using the first six K-L transformed 
components after textural augmentation 


Figure 6—Confusion matrix between classification maps 
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Figure 7—Distribution of channel values for spectral features 
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is that the features are defined here at every point in an 
image and the effective resolution is not reduced. An inter¬ 
pretation of the features extracted through linear combina¬ 
tions of the spectral reflectances of the nearest neighbors of 
a pixel is that the effects of the spatial variation in the 
immediate vicinity of the pixel are taken into account in 
making decisions on its classification. Experimental results 
are shown for a small subset of a Landsat scene using one 
of the feature definitions and the K-L transformation to 
reduce the dimensionality. The main differences between 
the classification map using spectral features and that using 
textural features are that the latter has a higher number of 
residential pixels and is smoother in appearance. An ex¬ 
amination of the plot of the spreads of the spectral and 
textural feature values for the various classes reveals better 
separability of classes with textural features. 

Further experiments with the features and dimensionality 
reducing transformations should include the use of symme¬ 
tric feature vectors such as in Equations (3) and (4) and the 
use of a canonical transformation. 

Combinations of three of the first few components ob¬ 
tained using the transformations can be used for producing 
color composites. Such color images will have an enhance¬ 
ment of the edges and are expected to be*similar to the 
output of high-emphasis filters. 
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INTRODUCTION 

JPL’s Image Processing Laboratory (IPL) has been pro¬ 
cessing imagery using digital computers for approximately 
ten years. During that period, the computer capability at the 
IPL has developed from a single job batch oriented system 
with no volatile image displays to a multi-task environment 
accommodating both batch processing and several interac¬ 
tive tasks simultaneously, supporting a variety of general 
purpose and special purpose interactive image display sys¬ 
tems. The sophistication of the various applications has also 
developed from simple subjective enhancement of planetary 
imagery to complex applications requiring correlation of re¬ 
motely sensed imagery with auxiliary data and with non¬ 
imaging data bases. This paper describes the evolution of 
the IPL facility, and illustrates the increasing complexity of 
the image processing tasks with examples of various appli¬ 
cations from the planetary program and the earth resources 
activities that have utilized the IPL facility. 


HISTORICAL BACKGROUND 

Ten years ago at JPL, digital image processing of lunar 
images from Ranger and Surveyor was performed using a 
central general purpose computer in a batch environment. 
An analyst submitted his job and often had to wait several 
days to view the result, since the job had to be run in 
competition with other tasks using the same computer and 
the output image had to be recorded on film, processed and 
printed before he could see the results of processing. 

In 1968, IPL acquired an IBM 360/44 computer system 
that was dedicated to image processing applications. This 
computer, and the 44 PS operating system, were capable of 
supporting a single user at any given time; the analyst again 
had to wait to view the result of his processing until film 
recording, film processing and printing had occurred. At 
that time, there were no commercially available image dis¬ 
play systems that could be easily interfaced to a digital 
computer so the IPL developed a core-refreshed display 
(CRD) system. The CRD used the 360/44 core memory as 
its refresh memory, and the display was capable of display¬ 
ing a 256 picture element square image with 4 bits of inten¬ 


sity quantization. The core memory of the 360/44 was read 
every 1/30 of a second to refresh the video display at video 
rates. The CRD made it possible to display imagery as soon 
as it had been processed, although the display was limited 
to 256 square resolution and 4 bits of intensity information. 
It had the obvious disadvantage of precluding any processing 
while image display was occurring, since the 360/44 memory 
was totally committed to image display refresh buffering 
whenever an image was being viewed on the CRD. Never¬ 
theless, it represented the first interactive image processing 
capability at the IPL, and substantially shortened the turn¬ 
around time for subjective enhancement processing. 

The 360/44 operating system and hardware display support 
improved between 1968 and 1975. A new image display 
system employing lithicon storage tube refreshed display 
systems was implemented to support the Mariner 9 Mars 
orbital mission in 1972. By 1974, it was possible to run two 
or three jobs on the system in a timesharing mode, using 
foreground and background partitions. However, processing 
became severely limited by the constraints of the 360/44 
system as the size of the imagery returned by the planetary 
missions increased, as the number of images returned from 
each mission increased dramatically, and as large multi- 
spectral images became available from the LANDSAT sat¬ 
ellites. In 1975, a decision was made to upgrade the entire 
IPL computer configuration and associated display systems 
to meet the following objectives: (a) increase batch process¬ 
ing capacity by at least a factor of three, (b) enable simul¬ 
taneous interactive image processing to occur at the same 
time that batch processing was occurring, (c) enable at least 
two interactive image processing users to be active at any 
given time. 

The next section describes the current IPL computer and 
display configuration, and the remaining sections discuss a 
variety of applications for which the system has been used. 

CURRENT IPL CONFIGURATION 

The current central computer in the IPL is an IBM 360/65 
with a megabyte of memory, eight tape drives, and 900 
mbytes of on-line disk storage; 800 of the 900 mbytes of on¬ 
line storage consists of Memorex 3670 high speed disk 
drives. 
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The 360/65 operating system is OS/MVT, and TSO is used 
to support timesharing operations. The Informatics MARK 
IV data base management system is also available, and is 
used to catalog processed imagery produced by the IPL 
each day. In addition, MARK IV is used to maintain catalogs 
of auxiliary data relating to each image processed by the 
IPL. For example, an image of Mars recorded by the Viking 
Lander will have a variety of parameters associated with it, 
such as local Mars time at which the exposure was made, 
camera gain and offset settings, solar illumination angle at 
the time of exposure, etc. MARK IV is used to construct, 
update and search these catalogs. 

The interactive processing is supported by a variety of 
terminals and display systems. A Digital Equipment Cor¬ 
poration PDP 11/40 computer is interfaced to a channel on 
the 360 via a DX-1 IB channel interface. The PDP 11/40 sup¬ 
ports several dial-up and hard-wired interactive terminals, 
and three image display systems. The interactive terminals 
include both IMLAC graphics/CRT terminals and TI Silent 
700 terminals. The current IPL configuration is shown in 
Figure 1. 

The image display systems are all commercially available 


equipment. They include a RAMTEK display system that 
accommodates 6 bit black and white imagery up to 640x 512 
elements. The RAMTEK also has two one-bit graphics 
overlay planes and two trackball/cursor units. Each of the 
RAMTEK output signals (video only, video plus either 
graphics plane, or either of the graphics planes) are available 
as video signals that may be separately accessed and dis¬ 
played. Thus more than one user can use the RAMTEK 
system at a given time. For example, one user can access 
the black and white video image with one graphics plane 
overlaid, while a second user can access just the second 
graphics plane. All video signals are routed into a video 
network that includes several black and white monitors dis¬ 
tributed in two user area locations at IPL, and a user selects 
the video signal he wishes to utilize and displays that signal 
on a monitor near his terminal. One view of one of the 
interactive user stations is shown in Figure 2, and a user 
interacting with an image display is shown in Figure 3. 

The other image display systems consist of two COMTAL 
display systems. A COMTAL 8003 system provides 
512x512 resolution for six bit images, and includes graphics 
overlay planes and trackball/cursor unit. The COMTAL 



Figure 1—JPL Image Processing Laboratory computer configuration 
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Figure 2—User area station at JPL’s Image Processing Laboratory. The 
user seated at the IMLAC terminal can select video signals to be routed to 
the display monitors using the video switching box on the right side. Audio 
communication with the computer operator is also provided. (JPL Photo 
Lab Negative No. 324-2009Ac) 


8003 provides either a single color image at 512x512 reso¬ 
lution or three black and white images at the same resolu¬ 
tion. Again, each output video signal is separately accessi¬ 
ble, and can be routed to any monitor in the user areas. 

A COMTAL 1024 system provides capability for display 
of black and white imagery at 1024x1024 resolution. Be¬ 
cause of the high bandwidth, the video output of this device 
is routed to a single specially modified monitor. The 1024 
system also includes graphics overlay and trackball/cursor 
unit. 

A complete description of the IPL computer configuration 
and the interactive capabilities supported by the PDP 11/40 
computer is contained in Reference 1. 



Figure 3—Interactive extraction of lakes from LANDSAT imagery 
achieved through utilization of the COMTAL display and associated 
trackball/cursor unit. (JPL Photo Lab Negative No. 324-2287Ac) 


GEOMETRIC TRANSFORMATION 

IPL has been performing digital geometric transformation 
of imagery for approximately ten years. The earliest appli¬ 
cations included removal of camera system induced geo¬ 
metric distortion 2-4 and projection of planetary imagery to 
standard cartographic projection. 5 More recently, geometric 
transformation has become an important tool in registration 
of imagery and non-imaging data represented as imagery to 
standard georeferenced coordinate systems, and in registra¬ 
tion of multiple images of the same surface area recorded at 
different times. 

Geometric transformation of digital imagery begins by 
locating a set of tiepoints within the original image and 
defining the locations of those tiepoints in the output image. 
The transformation of the tiepoints may be determined by 
a mathematical algorithm or by analysis of the imagery itself. 
For example, image transformation to standard mapping 
projections can be defined by establishing a uniform grid of 
tiepoints in the output image aligned with a longitude/latitude 
coordinate system, and then mathematically computing the 
locations of those tiepoints in the untransformed image 
based on a model of the spacecraft location and imaging 
sensor geometric characteristics. The locations in the input 
image corresponding to points between tiepoints in the out¬ 
put image is computed by an interpolation procedure. 

The intensity of picture elements in the output image that 
do not fall on exact points in the input image is computed 
by performing an intensity interpolation (or resampling) in 
the input image. The process is shown in Figure 4. Normally, 
the intensity of each sample on each line of the output image 
is computed in turn, starting at the upper left comer of the 
image and working down to the lower right comer. For each 
output sample, the corresponding location in the input image 
is computed, and the intensity of that sample is computed 
based on an interpolation scheme if the location in the input 
image is not exactly at a discrete image sample point. Typ¬ 
ical interpolation schemes include nearest neighbor, bilinear 
interpolation, and cubic spline. 

An example of geometric transformation is shown in Fig¬ 
ure 5. A Viking Orbiter image of Mars is shown before and 
after orthographic projection. The unprojected image shows 
the large crater to have an elliptical shape, due to distortion 



INPUT IMAGE OUTPUT IMAGE 


(x) = TIEPOINTS 

PICTURE ELEMENT [x] CORRESPONDS TO LOCATION Q IN THE INPUT IMAGE 

Figure 4—Digital geometric transformation 
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Figure 5—Left—Contrast enhanced version of a Viking Orbiter image of a 
large Martian crater. The crater is ellipsoidal, due to spacecraft viewing 
geometry. 



Figure 5—Right—The same image after orthographic projection. Shape 
distortion due to viewing geometry has been removed. (IPL Pic ID 
76/09/09/173047 and 76/09/09/175425) 


introduced because the image was recorded when the cam¬ 
era was at an oblique angle relative to the local surface 
normal. The orthographic projection is performed to depict 
the images as if the camera were viewing the scene from 
directly above the center of the field of view, and the trans¬ 
formation is defined based on spacecraft position and cam¬ 
era orientation data returned in the spacecraft telemetry. 

AUTOMATED IMAGE REGISTRATION 

It has frequently been possible to perform image registra¬ 
tion in an automated manner in the NASA planetary pro¬ 
gram. Typically, each planetary spacecraft records a large 
number of images of a planetary surface, and the science 
investigators require construction of photomosaics showing 
large portions of the surface of a planet in a single photo¬ 
graphic product. In most cases, it is possible to perform 
standard mapping projections by computing the geometric 
transformation for each individual image based on auxiliary 
data returned with each image that describes the spacecraft 
position relative to the planet and the camera viewing angle. 
The transformation is computed based on the mathematical 
model of the planet (generally an oblate spheroid is used as 
the planet model), the camera system optical components, 
and the spacecraft and camera viewing geometry. 

IPL produced over 1200 images projected to an ortho¬ 
graphic projection during the Mariner 9 Mars orbital mis¬ 
sion. Each image was individually corrected for camera sys¬ 
tem geometric distortion, enhanced, and then projected to 
an orthographic projection. The projected images were then 
photographically scaled and used to construct the first global 
photomosaic of a planet ever constructed from remotely 
sensed imagery. The photomosaic was constructed on a 


three-foot diameter globe, and a portion of the photomosaic 
is shown in Figure 6. 

The Mariner 10 flyby mission to Venus and Mercury pro¬ 
vided several thousand images of the surface of Mercury. 
IPL produced photomosaics of many standard mapping 
quadrants on the planet’s surface. Figure 7 shows one of the 
mapping quads produced from over fifty Mercury images 
from Mariner 10. Again each image was individually pro¬ 
cessed to remove camera system induced distortion, en¬ 
hanced, and projected to Mercator projection. Other map- 



Figure 6—One hemisphere of the photomosaic globe of Mars produced 
using over 1200 enhanced and orthographically projected images from the 
Mariner 9 Mars orbital mission. (JPL Photo Lab Negative No. 320-284B) 
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Figure 7—Computer generated mosaic of the region near the south pole of 
Mercury, constructed from over twenty separate Mariner 10 vidicon 
images. A special scene dependent high pass filter has been applied to bring 
out local detail throughout the scene, and especially near the terminator. 
(JPL Photo Lab Negative No. 324-2306) 


ping projections in standard use at IPL include Lambert 
Conformal Conic and Polar Stereographic. 

In both of the cases described above, the navigation data 
was of sufficient quality to enable mosaicking of automati¬ 
cally projected imagery. After each individual image was 
projected, it was not necessary to refine the projection trans¬ 
formation to achieve a better match of the imagery. How¬ 
ever, there are instances where precise registration to less 
than a picture element across the entire area of interest is 
required. In those cases, it is often necessary to perform 
manual or interactive processing to refine the transforma¬ 
tions that are performed. Typical applications of this type 
are discussed in the next section. 

SEMI-AUTOMATED IMAGE REGISTRATION 

IPL has developed the capability to perform interactive 
refinement of image registration for specific applications. 
An example of an application requiring registration accura¬ 
cies of less than one picture element is image differencing. 
For example, the Mariner 9 Mars Orbiter provided repetitive 
coverage of portions of the Martian surface during 1972. 
Determination of surface feature variation was one scientific 
objective of that mission. It was possible to achieve gross 
(accuracy ^1 picture element) registration of two images of 
the same portion of the surface taken at different times 
during the mission by performing standard mapping trans¬ 
formations. However, change detection can be confusing if 
there is misregistration on the order of one picture element, 
since changes are masked by overall misregistration when 
a difference image is produced. It was necessary to refine 
the registration using an interactive display and supporting 
software. The software originally developed for Mariner 9 
has been extended and refined for later applications, but the 
general principles remain the same. 

An analyst selects a feature in one of the two images he 


wishes to register. He then identifies the same feature in the 
second image. The program uses his second location in the 
second image as the center of a search area, and performs 
correlation calculations within a local moving window in the 
second image. A point of maximum correlation is selected 
as the location in the second image that corresponds to the 
selected location in the first image, and the offset of that 
point relative to the location in the first image is retained. 
The analyst proceeds to select a set of features and performs 
the same routine with each point, so that a set of offset 
vectors is ultimately available throughout most of the region 
of overlap. When an adequate number of feature offsets has 
been acquired, a geometric transformation is defined that 
will map the second image onto the first image. Normally, 
if the two images have been processed to a standard mapping 
projection prior to the identification of feature points, the 
second geometric transformation is a small refinement. The 
result is a registered pair of images where registration has 
been achieved to less than one picture element accuracy. 

Figure 8 shows one of the output products from the Mar¬ 
iner 9 image differencing task at IPL. The top two images 
are segments of Mariner 9 imagery of the same part of the 
surface recorded approximately two weeks apart. The lower 
left image is an unenhanced difference picture, and the lower 
right image is a contrast stretched version of the difference 
picture. Any differences in intensity between the two images 
is displayed as either whiter or blacker than mid-gray in the 
difference image. 

If the two images had been misregistered by more than a 
picture element, all features in the scene would appear in 
the difference image as both black and white detail. For 
example, the crater edge would appear as a double image, 
one black and the other white. Because the registration is 
precise in the example shown in Figure 8, the crater edge 
appears only once in the difference image, and the difference 
image correctly portrays the change in crater edge definition 
and albedo that occurred as the great Martian dust storm of 
1972 continued to clear during the two-week period in which 
these images were recorded. 

REGISTRATION OF IMAGING AND NON-IMAGING 

DATA 

The registration of non-imaging data to imaging data has 
always been important in digital image processing applica¬ 
tions at IPL. Processing of planetary imagery often required 
knowledge of auxiliary data relating to each recorded-image. 
For example, generation of geometric transformation param¬ 
eters to transform an image into a standard mapping projec¬ 
tion requires a knowledge of the spacecraft position and the 
camera viewing angle at the time the image is recorded. It 
is also often desirable to remove the shading effects in plan¬ 
etary imagery that are caused by variation in solar illumi¬ 
nation and viewing angles from image to image, and removal 
of these effects is performed using auxiliary ephemeris in¬ 
formation and spacecraft position and view angle data. 

The planetary program initially provided the motivation 
for geographically referenced image data bases. It was nec- 
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MARINER 9 ***B****** YR 72 BAY 355 GMT 221401 BAS TIME 04292410 

PICTURE NUMBER *27*** EXP TIME 6 MSEC FILTER PDS * 

VIEW ZENITH ANGLE **** SOLAR ZENITH ANGLE **** PHASE ANGLE **** 
*** ORTHOGRAPHIC PROJECTION *** 

AT PRO J. CENTER L* 500.0? S* 500.0? LAT=~60.5? LONG* 346.6 U 
SCALE 0.129 KM-PXLj NORTH* -94.00 BEG CLOCKWISE FROM UP 
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Figure 8—An example of image differencing. Top two images are segments of Mariner 9 images of Mars recorded approximately two weeks apart during 
clearing of the great dust storm of 1972. The two segments are in precise registration. The bottom left image is an unenhanced difference image, and the 
bottom right image is a contrast enhanced difference image. Changes in albedo in the center of the crater show as white areas in the difference image. (JPL 

Photo Lab Negative No. 324-2293A) 
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essary to index each image based on the longitude and lat¬ 
itude coordinates of the surface area recorded in each indi¬ 
vidual image. Computerized data bases were created for 
imagery of Mars, Venus and Mercury that enabled interac¬ 
tive interrogation to determine surface coverage as a func¬ 
tion of a variety of parameters. It is possible at IPL to 
perform a search to locate all images taken of a particular 
portion of the planet surface that were taken under a partic¬ 
ular set of conditions. Thus an analyst can interactively 
query a picture catalog to answer the following type of 
question: 

“What Viking Orbiter images exist that show the area 
within 10 degrees of latitude 65° west longitude 120° after 
3:00 p.m. local Martian time that were taken through the 
red spectral filter? Limit the search to those images re¬ 
corded after the 35th day of the Orbital mission.” 

The LANDSAT satellites have increased the emphasis on 


correlation of geographically encoded data bases. There are 
literally hundreds of existing geographically encoded data 
bases in routine use throughout the country by a variety of 
federal, state and local government agencies. One of the 
most familiar examples is the data base maintained by the 
United States Census Bureau. LANDSAT satellites provide 
repetitive multispectral imagery of the earth’s surface, and 
have been used to monitor agricultural parameters, geolog¬ 
ical properties, and land use patterns throughout the coun¬ 
try, to list just a few applications. In almost every case, 
there is a need to relate results obtained from analysis of 
LANDSAT imagery to existing geographically referenced 
data bases. In recognition of this need, IPL has developed 
the Image Based Information System (IBIS). 6,7 This system 
utilizes the geometric transformation capabilities originally 
developed for the planetary program to correlate imaging 
and non-imaging data bases. Figure 9 shows the results of 
a standard thematic classification of land use in the Portland, 
Oregon area. 9 Each picture element in a LANDSAT multi- 



PORTLANDj OREGON 

CENSUS TRACT 
LANDSAT CLASSIFICATION 


■ RESIDENTIAL 

■ COMMERCIAL 

& INDUSTRIAL 

■ OPEN SPACE 
E3 AGRICULTURAL 
f~1 FOREST 

□ WATER 

S 1 SEASONAL CHANGE 


APRIL ✓ JULY ±976 
0 5 10 15 

KILOMETERS 


Figure 9—Thematic map of Portland, Oregon showing land use patterns determined from multispectral classification of LANDSAT imagery. The white overlay 

indicates U. S. Census tract boundaries registered with the LANDSAT image. 
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Figure 10—District image created from tabular data base containing traffic 
zone boundaries in Portland, Oregon. The district image display product is 
created from the digital district image using a five-color algorithm; 9 the 
district image is registered to the LANDSAT thematic map shown in 
Figure 9. 


spectral image is assigned to a particular class of land use 
based on the relative spectral response in each of the four 
spectral bands recorded by LANDSAT. Training areas for 
which ground truth is known are often used to define the 
spectral signature corresponding to each desired land use 
category. 

The graphical overlay visible in Figure 9 represents the 


boundaries of the Census tracts for the Portland area. The 
Census information has been registered to the LANDSAT 
image using the same registration techniques previously de¬ 
scribed for the planetary applications. The U. S. Census 
Bureau provides geographically indexed graphical files on 
magnetic tape. The data base provided by the Census Bu¬ 
reau includes the geographic location of each of the vertices 
visible in Figure 9, and a set of parameters for each census 
tract (e.g., population, age distribution). The Census tabular 
file is read, and a district image is created. The district image 
is registered to the LANDSAT image, and contains picture 
elements that are coded to indicate the census tract identi¬ 
fication number in which that picture element lies. An image 
display product of the district image is often created as a 
visual check on the district image. Figure 10 shows a district 
image for the Portland area, based on traffic zone definitions 
within the city of Portland rather than the Census data. It 
is important to note that often several overlapping geocoded 
data bases exist for a single area. In the case of the Portland 
application, we see that both Census data and traffic zone 
data bases exist, and that the boundaries of the traffic zones 
do not correspond to Census tract boundaries. This is one 
reason that an image-based correlation procedure is useful. 
It is possible to perform cross-correlations between multiple 
overlapping geographically referenced data bases and re¬ 
motely sensed multispectral imagery using the IBIS system; 
the tabulation is achieved by simple line by line summation 
using the multiple registered imagery. As an example, Figure 
11 contains a portion of a tabulation of land use by traffic 
zone in Portland. The land use data is obtained from the 
thematic map shown in Figure 9, and the tabulation by traffic 
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Figure 11—Tabulation of land use by traffic zone in Portland, created by using a registered district image in conjunction with the thematic map. 
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zone is achieved by correlation of the traffic zone district 
image with the thematic map. 

The IBIS system has recently taken advantage of the 
interactive capabilities of the current IPL computer system. 
In particular, the image display systems provide a conve¬ 
nient way to perform precise registration of non-imaging and 
imaging data bases. As an example, the Census Bureau 
tabular data base often contains errors, resulting in failure 
of tract boundary lines to meet at vertices or generation of 
erroneous tract boundary lines and an erroneous district 
image when the tabular data is converted to image format. 
An interactive display system with graphics overlay capa¬ 
bility provides an efficient tool for editing of the census data 
base to correct or remove occasional erroneous data entries. 

SUMMARY 

This paper has provided an overview of the evolution of the 
computer configuration at JPL’s Image Processing Labora¬ 
tory, and a summary of the development of techniques for 
geometric transformation of digital imagery. Image registra¬ 
tion techniques originally developed for the planetary pro¬ 
gram have been described, and have become an important 
component of the registration of non-imaging geographically 
encoded data bases with multispectral imagery returned by 
earth observations satellites. Registration of existing geo- 
coded data bases with LANDSAT imagery will continue to 
be important if the LANDSAT data is to be truly useful to 
the user community. 
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INTRODUCTION 

With two LANDSAT satellites in prbit and a third to be 
launched in the near future, the use of imagery obtained by 
satellite for monitoring the earth’s resources is well estab¬ 
lished. The timely, efficient, and routine handling of this 
image data is not so well established, however, and large 
systems for this purpose are just now comihg into being. 
Two examples of such systems are NASA’s MDP system 
being developed by IBM, and Department of the Interior 
EROS Data Center’s EDIPS system being developed by 
TRW. Under consideration for the future is a system for 
handling imagery from LANDSAT D, whose data volume 
and throughput requirements are substantially more severe. 
This paper discusses some of the design considerations 
forced by the requirements of such systems and suggests 
solutions. 

The main driving requirements for systems processing 
LANDSAT data are the extremely large data volume and 
the high throughput necessary to handle the data in a timely 
manner. These two requirements also separate the design of 
these systems from normal data handling systems which 
deal with lower volumes and throughput. The nature of the 
processing performed on LANDSAT data also differs from 
the processing performed in other applications. Special 
processing may include geometric warping of the imagery, 
radiometric correction, edge enhancement, and manual dis¬ 
play interaction. Finally, the output products of these sys¬ 
tems include annotated image films, and digital tapes of the 
imagery for specific users. The data volume and throughput, 
special processing, and special output products of LAND¬ 
SAT data all press for unique solutions not found in other 
data processing systems. 

Due to the high throughput and variety of special proc¬ 
essing to be performed on the data, a pipelined design for 
the system is frequently the best choice. With a pipelined 
system, data from one scene may be read into the system 
while data from another scene is having special processing 
performed, and yet another scene is being output to film. 
Pipelining here results in maximizing system resource utili¬ 
zation and throughput while minimizing the total amount of 
hardware required. It is thus an efficient, high throughput, 
and cost effective design. Pipelining at a low level may mean 


the use of pipelined signal processors or microprocessor 
hardware. Pipelining at a high level involves the use of 
several general purpose computers, each of which is dedi¬ 
cated to performing a processing segment in the pipeline 
flow, in parallel with the other computers. What level of 
pipelining is best depends on the specific application and the 
hardware available. Figure 1 shows the processing flow of 
a typical pipelined system. 

The next section discusses the specifics of LANDSAT 
image systems requirements; the third section briefly dis¬ 
cusses special processing performed on the data; the fourth 
section discusses key design considerations and suggested 
solutions; and the last section summarizes the key points. 


SPECIFIC LANDSAT IMAGE SYSTEM 

REQUIREMENTS 

LANDSAT image processing system requirements are 
driven by the image sensors aboard the spacecraft. 1 For 
LANDSAT’s 1, 2, there are two sensors: The Multispectral 
Scanner (MSS) and a set of Return Beam Vidicons (RBV). 
The vidicons have not been operational since early in the 
mission. On LANDSAT C there is an MSS and a pair of 
higher resolution RBV’s. The MSS has 26 detectors arrayed 
to image four spectral bands with a resolution of 79 meters 
and a fifth band with a resolution of 237 meters. In geo¬ 
metrically correcting the MSS scenes, the images are over¬ 
sampled so that each picture element (pixel) is 57 meters 
square. An MSS scene is 185 kilometers square, so that an 
output image is 4V9 bands of about 3300 2 pixels. A similar 
computation yields a LANDSAT C RBV scene of about 
5300 2 pixels, but only one band. A complete MSS scene is 
imaged roughly every 30 seconds, and the sensor is used 
over one hour per day. Combined data from LANDSAT’s 
1, 2, and C totals about 200 MSS scenes and 160 RBV scenes 
per day. Thus, a typical system requirement is to process 
this data in a two-shift working day. Total data volume is 
about 14 billion bytes per day. Allowing for gaps, spacecraft 
telemetry data, and annotation data (added in later process¬ 
ing) on the high density tapes on which the data is typically 
recorded, the system must be able to process data at a 
continuous rate of about 500,000 bytes per second. 
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Figure 1—Processing flow of a typical pipelined Landsat image processing 
system 


In addition to throughput requirements, LANDSAT image 
processing has accuracy requirements on the special geo¬ 
metric and radiometric correction processing performed on 
the data. Typical requirements are that the output pixels be 
accurate to within one gray level (eight bit data having 256 
gray levels) after radiometric and geometric correction, and 
that the geometric correction algorithm used introduce no 
more than Vio of a pixel spacing error (i.e., Vio of 57 meters 
for the earlier LANDSAT’s) in the location of the output 
pixels. Further, in the case of geometrically correcting the 
data to register it to other LANDSAT images or to control 
points on the earth with known locations, the position error 
of output pixels may be required to be less than Vi or Vi 
pixel spacing. These accuracy requirements impose strin¬ 
gent conditions on the algorithms used, more than on the 
system design. However, the overall system design may 
well be impacted by the increased complexity and compu¬ 
tation time of the sophisticated algorithms necessary to meet 
the accuracy requirements. 

Reliability requirements for this type of system cannot be 
overlooked in the system design. System availability re¬ 
quirements of 85-90 percent are typical. Hardware redun¬ 
dancy, switchable peripherals, hot spares, and modular re¬ 
placement of hardware components at the board level are 
all potential methods for meeting the reliability requirements 
and must be considered in the selection of a system design. 
Software reliability is also important. Detailed fail-safe error 
checking by the software modules, top-down design, and 
structured programming can and do help to insure reliable 
software, and are considered in the design and development 
of the system. 

Although requirements on accuracy, reliability, special 
processing, etc., are important, the throughput requirement 
remains the most critical in selecting a system design. How 
these requirements determine the key issues in the system 
design will be discussed in the fourth section. 


SPECIAL PROCESSING 

The nature of LANDSAT imagery frequently requires 
special processing on the image data. This section briefly 
discusses some of the special processing, including geomet¬ 


ric warping, radiometric correction, image enhancement, and 
format conversion of the input data. Although not all of these 
processes are required for every system, they are typical of 
the kinds of processing done on LANDSAT data and fre¬ 
quently drive aspects of the system design. 


Geometric correction 

The characteristics of the spacecraft, earth, and imaging 
sensor combine to distort the true ground data into the raw 
image data received from the spacecraft. This raw data must 
be corrected to show the true ground picture. The effects of 
earth rotation, spacecraft position and attitude, sensor scan¬ 
ning nonlinearities, earth curvature, and time delay between 
adjacent detector’s exposures must all be removed by the 
geometric correction processing. In addition, the output 
products are frequently required to be in one of several map 
projections, the most common being the Universal Trans¬ 
verse Mercator (UTM 2 ). Finally, the image data may be 
required to be registered to an earlier scene or to ground 
control points of known location. These effects are also 
allowed for in the geometric correction processing. 

The geometric correction processing may proceed in two 
stages (which may be two segments of the pipeline). The 
first is the generation of a set of values at a grid of regularly 
spaced locations throughout the corrected image. The values 
specify the location of the corresponding pixels in the input, 
uncorrected image. The accuracy requirement drives the 
number of grid points, which in turn determines the com¬ 
putation and storage load for this portion of the processing. 
This grid is generated from the equations for the distortion 
effects mentioned above, the most significant being the 
spacecraft attitude and the map projection. The equations 
are complicated and their computation time must be care¬ 
fully estimated. 

Once the grid of locations is determined, the warping itself 
must be performed. Each pixel of the output image is located 
by interpolation from the grid of input locations in the input 
image to a fractional pixel location. The output pixel value 
is then interpolated from the values of its input pixel neigh¬ 
bors. 

The complexity of the warping algorithm and its para¬ 
metric nature generally preclude its implementation purely 
in hardware, but the right choice of a hardware-software 
mix is not always obvious. The generation of the grid of 
location values in the first stage of the geometric correction 
involves computation of complex equations and is usually 
done in a general purpose computer. The nature of these 
computations, however, does impact the computer selec¬ 
tion. 


Radiometric correction and image enhancement 

For the MSS sensor (and the TM sensor to be on LAND¬ 
SAT D) there are multiple detectors with different charac¬ 
teristics. Therefore, different detectors will give different 
outputs for the same input radiances. Removing the detector 
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specific errors in the pixel values is known as radiometric 
correction. This may be done by constructing a table for 
each detector which maps the pixel values received from 
the spacecraft into radiometrically correct values. Thus, for 
LANDSAT’s 1, 2, and C there would be 26 tables of 64 
elements each (since the data as received is 6 bits per pixel). 
As with geometric correction, this proceeds in two stages 
(which again, may be separate segments of the pipline). The 
first stage is the construction of the detector-specific tables 
themselves, and the second is the mapping of the pixels 
through the tables. Construction of the tables may require 
obtaining statistics about the image data in the form of mean 
and variance or histograms of the pixel values or calibration 
data in order to remove image striping caused by the differ¬ 
ences between detectors. In a general purpose computer, 
computation of these statistics requires several microsec¬ 
onds per pixel; hence, special purpose hardware or firm¬ 
ware is frequently dictated by the requirement to construct 
these tables. 

Once the tables are constructed, the pixels must be 
mapped through the tables to generate the radiometrically 
corrected values. Since this again requires handling each 
pixel, special purpose hardware may be the solution to map¬ 
ping the pixels at the required throughput. 

Image enhancement may involve filtering the input (cor¬ 
rected or uncorrected) image to enhance edges, or remove 
haze or contrast stretching the image so that scenes of low 
contrast look better visually. These processes may take ad¬ 
vantage of the scene statistics gathered at an earlier stage of 
the pipeline. Depending on the complexity of the algorithms 


used, special purpose hardware or firmware may be used, 
but again, normally a general purpose computer lacks suf¬ 
ficient speed to handle these operations. Figure 2 shows a 
corrected unenhanced image compared to the same image 
after edge enhancement. 

Format conversion 

LANDSAT image data as received from the satellite is 
interleaved by detector in a formal called “band-interleaved- 
by-pixel” (BIP), i.e., one pixel is received from each detec¬ 
tor before the second pixel is received from each detector. 
Since an image line is formed by one detector as it scans 
the earth, in order to reform the image into image lines, the 
data format must be converted first into lines (band-inter- 
leaved-by-line or BIL) and finally into band-sequential 
(BSQ) where all lines of an MSS (or TM) spectral band are 
consecutive. Converting from BIP to BIL again involves 
handling each pixel, but only moving it from one location to 
another. Thus, specially addressable memories or computer 
controlled direct memory access devices (DMA’s) may be 
the solution here. For 26 detectors of 2300 pixels per input 
line, a memory of 60,000 bytes would be required—quite 
reasonable for modern minicomputers. Conversion from 
BIL to BSQ, however, could require a memory large enough 
to hold an entire scene (31 million bytes for the earlier 
LANDSAT’s) unless enough devices were available in the 
next pipeline segment to handle all bands simultaneously. 
Thus, a mass storage device such as a disk is usually needed 
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here. Formatting this storage device so that image data is 
written on in BIL and later read off in BSQ, both at the 
required data rates, is a nontrivial task. Disk formatting will 
be discussed in more detail in the next section. 

The special processing described in this section often ne¬ 
cessitates special purpose hardware or firmware for handling 
the individual pixels. Even where pixels are not handled 
directly, the algorithms involved may impact hardware se¬ 
lection. The next section discusses further these and other 
key design issues. 

KEY DESIGN ISSUES 

The requirements of throughput, accuracy, reliability, 
economics, and special processing described above, force 
the selection of a system design. This is not to say that there 
is only one solution to these problems, but that there are 
certain key design issues which must be dealt with. Some 
of these, such as the choice of special hardware and memory 
sizing have been mentioned above. Other issues are the 
selection of general purpose computers, the need for a proc¬ 
ess control scheduler to handle the pipeline flow, efficient 
use of mass storage devices, and software development 
methodology. Obviously all of these cannot be discussed in 
depth here due to lack of space; however, this section looks 
briefly at each issue, showing the problem, and suggesting 
a method or methods for its solution. 

System loading considerations 

The throughput requirement of hundreds of kilobytes per 
second frequently means many transfers per second through 
a general purpose computer’s memory, perhaps several 
times the overall throughput rate. For example, suppose the 
system of Figure 1 were to be implemented on one general 
purpose computer. The transfers shown in Figure 3 may 
then be executed for each scene. Although the pipeline seg¬ 
ments’ processing occurs sequentially for a given scene, 
normally the pipeline will be kept full and all of these seg¬ 
ments will be executing on different scenes simultaneously. 

In this example, ten transfers to and from memory are 
executed per pixel. At a data rate of, say 700,000 bytes per 
second on the average, this means an average data rate of 



7 million bytes per second. In practice, the actual peak data 
rate may be substantially higher than this. For example, 
although the high density tape (HDT), film generator, and 
array processor transfer at a fairly constant rate, the disk 
and tape transfers occur much faster while actually perform¬ 
ing the record transfers than the average rate which includes 
inter-record gaps. Thus, the transfers 2, 3, 7, 8, and 10 may 
reach 1.2 million bytes per second each. This gives a total 
peak rate of 9.5 million bytes per second (or actually more 
since the CPU normally fetches instructions and data from 
this memory as well). Thus, system loading considerations 
may force the division of the flow into specific pipeline 
segments independent of the allocation of processing func¬ 
tions to segments. This requires a memory with an access 
time of about 100 nanoseconds per byte. Furthermore, in 
the case of many minicomputers (e.g., PDP-11, ECLIPSE, 
SEL 32, etc.), the data must travel via one memory bus; 
hence, this bus must be capable of sustaining the data rates 
calculated here. 

One potential solution to this problem is the selection of 
a computer with several data busses or even high speed I/O 
channels dedicated to transfers between certain peripherals 
and memory. Still another solution might be the selection of 
several banks of memory, each with its own memory con¬ 
troller to give independent parallel access to each bank. 
With this solution, (assuming the memory bus or busses 
could handle the data rate) buffers could be placed in dif¬ 
ferent memory banks to reduce the load on each individual 
bank to an acceptable level. A variation on this solution 
would be to interleave the memory banks with a least-sig¬ 
nificant-bit addressing scheme to reduce the effective access 
time of the memory. This has a particular advantage here 
since most transfers are of image lines which transfer from 
successive locations in memory. A final solution might be 
to construct the process control software described later to 
preclude processes which strain the system’s memory or 
bus capacity from running in parallel. If all of these potential 
solutions fail the analysis, more computers may be used, 
even to the ultimate extent of having one computer per 
pipeline segment. (Unfortunately, this may dictate still an¬ 
other computer to provide process control for the pipeline 
flow.) 

A further problem occurs in considering the computational 
load on the computer’s central processing unit (CPU). This 
is related directly to the software used by the system. 

Current minicomputer operating systems, for example, 
have software overhead times of Vz millisecond per inter¬ 
rupt. If the system of Figure 3 receives an interrupt on 
completion of the transfer of every image line, there will be 
an interrupt about every 4 milliseconds for each of the ten 
transfers. This allows .4 msec for processing each interrupt, 
which may well be beyond the capacity of the system. Po¬ 
tential solutions to this are to get a new, faster operating 
system or to write device handlers which speed the interrupt 
processing or even trigger the start of the next transfer when 
the current one is complete. Another measure would be to 
generate an interrupt only after the transfer of several lines. 
Whether this is possible depends on the image data formats 
and the peripheral and computer hardware. 
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Hardware /software /firmware tradeoffs 

The variety of special processing being performed on 
image data requires a mixture of general purpose computers 
and their software, firmware in the form of microprocessors 
or array processors, and special purpose hardware. In nearly 
all large scale, pipelined systems there will exist a need for 
a process controller. Due to the complexity of this scheduler 
and the need to control the various peripherals, a general 
purpose computer will normally form the heart of the sys¬ 
tem. This computer may also be well-suited for interacting 
with operators and users via displays or terminals. In addi¬ 
tion, it may perform some complex, low throughput special 
processing such as the generation of the grid of geometric 
correction locations or the radiometric correction tables 
from the scene statistics described earlier. 

However, a general purpose computer is not normally 
suited for processing that deals with pixels on an individual 
basis, due to a lack of the speed required by such processing. 
Thus, either special firmware of hardware must be em¬ 
ployed. The choice of firmware or hardware depends pri¬ 
marily on the complexity of the processing and the through¬ 
put required. As an example, the more complex functions 
of geometric warping and edge enhancement .would probably 
be done in firmware if the required throughput could be met, 
while the format conversion and contrast stretching could 
be done in hardware. The gathering of scene statistics and 
radiometric correction could be done in either hardware or 
firmware, depending on cost effectiveness. Very complex 
algorithms such as geometric correction could be done in 
commercially available array processors (such as Floating 
Point Systems AP-120B, Signal Processing Systems SPS-81, 
or CSP, Inc.’s MAP-300). If necessary to meet the through¬ 
put requirements, an image can be split into parallel streams, 
each passing through a separate array processor with the 
output being merged again (if the algorithm allows such a 
split). These array processors are sufficiently complex that 
configuring them with the proper memory and most suitable 
options raises design issues on its own. 

The decision to use special hardware or firmware in var¬ 
ious parts of the system may also force segmentation of the 
pipeline at specific points. This and cost-effectiveness con¬ 
siderations make this a key design issue. 

Process control 

One disadvantage of a pipeline system design is the need 
for a more complex process control program or scheduler. 
On a strictly sequential system resource conflicts do not 
usually occur since only one process is in progress at any 
time. Similarly, a totally parallel system in which all re¬ 
sources are duplicated does not normally give resource con¬ 
flicts. However, in even a simple pipelined system such as 
that of Figure 1, resource conflicts do occur, and the sched¬ 
uling of the pipeline processing segments must be done. In 
Figure 1, for example, several resource conflicts are appar¬ 
ent: Scene 2 cannot begin its image corrections immediately 
after input because the array processor required for it is in 


use for Scene 1; similarly, conflicts arise in the case of the 
output film generation hardware. Other conflicts may arise 
not in the use of specific hardware, but in the total system 
load. If several special processing segments had the hard¬ 
ware to execute simultaneously, they might still exceed the 
load capacity of the system as far as memory or bus loads 
are concerned. Further, the scheduler may need to prevent 
other processes from executing concurrently because of 
conflicts in accessing shared data areas. In short, in a pipe¬ 
lined system a scheduler may be necessary to eliminate 
resource conflicts. (If the processing flow is static and the 
same for all scenes, the scheduler may be very simple or 
even nonexistent in that the individual pipeline segments tell 
each other when they are complete or available; but for 
more complex systems, the scheduler may become quite 
involved.) 

The main functions of a scheduler in this type of system 
are: to determine the processing flow for each scene, initiate 
processes, determine processing conflicts and resolve them, 
and to monitor the processes for termination. All processes 
can communicate with the scheduler to indicate termination 
or error conditions, and in general need not communicate 
directly with other processes. Other scheduler functions 
may include operator interaction to determine each scene’s 
processing flow and to report error conditions. If the system 
runs out of resources, the scheduler may stop the flow 
altogether as the pipeline “backs up.” For example, in Fig¬ 
ure 1, if the film output generator fails, later scenes will be 
unable to complete their processing which means that they 
must be stored on mass storage devices. As these devices 
fill up, the image correction, special processing, and input 
segments have no place to store their data, so that finally 
input must stop. If the input is from tape, data may not be 
lost; however, if the input is directly from the satellite, data 
may be lost at the input end of the system. (The system 
designers could, of course, construct the scheduler so that 
data is not output to film, but is saved on tape, or other 
measures to prevent the loss of data.) 

The internal design of a scheduler is beyond the scope of 
this paper, but a scheduler is one solution to the problem of 
process control, which is an important design consideration. 

Use of mass storage devices 

In order to keep the process running smoothly, it is fre¬ 
quently necessary to use mass storage devices such as disks 
as buffers between pipeline segments. Otherwise, lack of a 
needed resource for even a brief period may bring all proc¬ 
essing to a halt as main memory buffers fill up rapidly. In 
addition, disks may be necessary to store image data written 
in one order to be read in another order. Since industry 
standard 80- and 300-million byte moving-head disks are 
commercially available, these are good choices for mass 
storage buffers. Several issues arise in using these disks 
efficiently. 

The data formatting on the disk is a design tradeoff which 
must be made. Figure 4 shows a typical disk format for 
buffering MSS data. Each track of the disk is hardwired as 
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23 sectors of 768 bytes each. There are 5 tracks per cylinder. 
There is no time penalty for disk accesses which change 
tracks within a cylinder, but there is a substantial penalty 
for switching cylinders which requires a movement of the 
read/write head. If each image line is 3500 bytes, then 4 lines 
will fit into 19 sectors. Since the disk rotates at 3600 rpm, 
it takes about 14 msec to read 4 lines of data. Then 2 sectors 
are skipped to allow about 1 Vi msec for software overhead 
before the next 4 lines are read. Twelve sectors are left 
empty at the end of the cylinder to allow time for the head 
movement to switch to the next cylinder. Thus, the reading 
of 20 image lines requires on the average of 5 disk revolu¬ 
tions. This gives a sustained throughput capacity of 840,000 
bytes per second, which is in excess of the required 700,000. 
Hence, there is some spare time available for situations in 
which system loading prevents being able to read the next 
4 lines within IV 2 msec after the last 4 are completed. 

Because of the high speed involved, there is no time 
available for retries of disk reads or writes in the event of 
parity errors. However, with modern disks, the parity error 
rate is so low that the expected number of errors affects the 
image significantly. Thus, parity checking need not be per¬ 
formed. If parity checking is performed, retries should not 
be made, and a system alarm should be generated only if 
the error rate becomes too high. Parity checking, formatting, 
high speed software device handlers, and reliability are some 
of the key design issues in using mass storage devices effi¬ 
ciently. 

Software methodology 

In designing high throughput, reliable systems, the hard¬ 
ware design is not the only consideration. The software 
design and implementation for general purpose computers 
and for microprogrammable microprocessors and array proc¬ 
essors is also extremely important. Software design prob¬ 


lems arise in attaining the high throughput required, and in 
developing reliable software which is free of timing-depend¬ 
ent bugs. Further, as research progresses, new algorithms 
are developed, and new user needs become apparent, the 
requirements for LANDSAT image processing change. 
Therefore, the software should be modular enough to permit 
easy modification or inclusion of new processing. The 
scheduler should be sufficiently flexible that addition of new 
segments in the pipeline does not cause major redesign or 
recoding of the scheduler. 

One means of attaining the high throughput required in 
array processors is to actually code and benchmark the 
critical high speed inner loops of the algorithms in the design 
phase, rather than waiting until the coding phase. This way, 
a good estimate of the actual throughput of these processing 
segments can be obtained early. 

In developing reliable software, the top-down design proc¬ 
ess is very valuable. Software tools which aid this process 
are very beneficial in easily developing a top-level design 
and expanding it to deeper and deeper levels until the soft¬ 
ware can be coded directly from the design document. If 
done properly, the detailed design document also becomes 
the main software documentation. Two good tools for soft¬ 
ware design are structured flow charts and Program Design 
Language (PDL). Impressive cost savings in software de¬ 
velopment through the use of PDL have been cited. 3 Soft¬ 
ware design tools aimed at aiding pipelined systems design 
would be of great benefit. These tools would facilitate check¬ 
ing for deadlocks, use of common data areas, etc. Unfor¬ 
tunately, we know of no such tools available at the present 
time. 

Structured programming is also of benefit in developing 
software for pipelined systems, and especially for modifying 
it in the event of changing requirements. In using FOR¬ 
TRAN for the software development, the use of a structured 
FORTRAN preprocessor can be of benefit, especially when 
the design was done in a top-down structured fashion. 
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SUMMARY 

This paper has shown how the requirements of large scale 
LANDSAT image processing systems force the considera¬ 
tion of certain key design issues. The main requirements of 
the system are the large data volume and high throughput, 
reliability, need for special processing, and accuracy. Proc¬ 
essing peculiar to the LANDSAT systems includes geomet¬ 
ric and radiometric correction, image enhancement, and high 
speed data reformatting. A pipelined design is often a natural 
choice for these systems. Although there are many issues to 
be tackled in the design of such systems, only the most 
important ones have been addressed here. These include 


system loading considerations in selecting computer hard¬ 
ware; the decision on where to use special hardware or array 
processors; the need for process control to schedule the 
pipeline’s segments; the efficient use of mass storage de¬ 
vices; and software tools and techniques. 
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AOIPS—An interactive image processing system 
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INTRODUCTION 


Prior to the fall of 1976, computer aided data analysis sup¬ 
port for applications investigations at NASA’s Goddard 
Space Flight Center was conducted on large scale IBM Sys- 
tem/360 computers in a batch oriented environment without 
the aid of image display units. As a result, even simple image 
processing operations took from days to weeks to produce 
desired results and required numerous manual, errorprone 
steps to produce the desired end product. In addition, op¬ 
erations which required a high degree of human interaction 
with the image data during the information extraction proc¬ 
ess (e.g., derivation of wind fields from cloud motions in a 
time-lapsed series of satellite images) were not practical in 
the batch environment. 

The Atmospheric and Oceanographic Information Proc¬ 
essing System (AOIPS) was developed to help applications 
investigators perform required interactive image data anal¬ 
ysis processes rapidly and to eliminate the inefficiencies and 
problems encountered with the batch operation discussed 
above. The major AOIPS hardware components are shown 
in Figure 1. 

Initial hardware deliveries occurred in September 1975 
and consisted of a PDP-11/70 computer and peripherals and 
a modified General Electric Company Image 100 system, 
which included an interactive image analysis terminal and 
a PDP-11/45 computer. The Image 100 system was desig¬ 
nated as AOIPS Image Analysis Terminal 1 (IAT1). Two 
additional image analysis terminals, designed at Goddard 
Space Flight Center and implemented by Hazeltine Corpo¬ 
ration, were delivered in May of 1976 and May of 1977. 
These terminals have been designated as AOIPS Image 
Analysis Terminals 2 (IAT2) and 3 (IAT3). 

In parallel with the hardware development activities, a 
series of applications software packages was developed to 
implement specific information extraction processes to meet 
established image data analysis requirements. AOIPS op¬ 
erations during 1976 and 1977 have demonstrated its capa¬ 
bility to solve many operational image processing problems, 
and has greatly extended an investigator’s ability to perform 
image information extraction operations in a flexible, effi¬ 
cient manner. 


SYSTEM OVERVIEW 

The heart of the AOIPS is a Digital Equipment Corpora¬ 
tion PDP-11/70 computer to which three magnetic tape 
drives, three disk units and other standard peripherals are 
attached. The image analysis terminals interfaced to the 
PDP-11/70 central processing unit provide for user interac¬ 
tion in the extraction of information from digital images. 

Terminal 1 (see Figure 1) provides capabilities for time 
lapsed display of image data and for output of video signals 
to a video disk storage unit. Control of this terminal is 
provided by its own PDP-11/45 computer. Digital images on 
computer-compatible tapes are loaded into a 5-channel solid- 
state refresh memory from two 800/1600 bit-per-inch (bpi), 
9-track tape drives. The contents of the refresh memory 
channels are input to analysis console digital logic for ratio- 
ing, scaling, and limit comparison; the output of these op¬ 
erations is displayed on both color and black and white TV 
monitors. Using the analysis console logic, image enhance¬ 
ment and multispectral classification analyses are performed 
under control of software on the PDP-11/45. Terminal 1 is 
interfaced to the AOIPS PDP-11/70 computer through a dual 
port, shared-disk unit capable of formatted storage of 88 
megabytes (8 bits per byte) of data. 

Terminals 2 (see Figure 1) and 3 are identical and are 
interfaced directly to the PDP-11/70 through a high-speed 
direct memory access interface for control signals. Using 
the three 800/1600 bpi, 9-track tape drives of the PDP-11/70 
and the High Density Digital Tape (HDDT) unit, magnetic 
tapes are loaded into the 5-channel solid-state refresh mem¬ 
ories of Terminals 2 and 3. Images may also be loaded from 
disk or from maps or photographs digitized by a video input 
scanner interfaced to the terminal. The contents of the re¬ 
fresh memory channels pass through lookup tables, digital 
matrix switches, and other image manipulation logic in the 
console. Image enhancement and parameter extraction anal¬ 
yses are performed under control of software on the PDP- 
11/70. 

The video disk, which is connected to all three terminals 
through a video switching system, has the capacity to store 
up to 600 TV-sized images. Using the switching system, 
images may be recorded from or played back to the TV 
monitors of any of the terminals. 
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Figure 1—Atmospheric and Oceanographic Information Processing 
System—AOIPS 


The AOIPS PDP-11/70 computer communicates with a 
remotely located IBM S/360/91 computer through a 4800-bit- 
per-second telephone line connection. 

A Dicomed Image Recorder (shown in Figure 1) functions 
as an off-line device to convert information recorded on 
computer-compatible tapes (CCTs) into photo products of 
image and related ancillary data. 

AOIPS utilizes computer-compatible tapes of multispec- 
tral digital image data generated by satellite and aircraft 
image scanners as well as related digital ancillary data re¬ 
quired to perform specific information extraction operations. 


In the future, high density digital tapes will provide the 
primary input medium for meteorological satellite data. The 
system provides capabilities for inputting digital data in sev¬ 
eral formats; performing standard image preprocessing op¬ 
erations including image registration, geometric correction, 
scanner distortion correction, and zooming and reducing 
subimages; executing image processing functions including 
level slicing, contrast enhancement, pseudo color and false 
color combinations, band ratioing, linear combinations, and 
histogram generation; performing analytical operations in¬ 
cluding multispectral classification and special applications 
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analyses; and displaying and generating hard copy output of 
single images overlayed with data plots, contours and var¬ 
ious forms of image annotation. 


USER—SYSTEM INTERFACE 

There are two types of users supported on the AOIPS: 
(1) applications investigators who specify and perform in¬ 
formation extraction operations on the system for specific 
applications, and (2) programmer/analysts who develop new 
systems and applications software. Applications investiga¬ 
tors interface with the system by requesting execution of 
various system processes from menus presented on inter¬ 
active cathode ray tube display units. Adopting the menu 
approach as the primary user interface has simplified train¬ 
ing problems and virtually eliminated the requirement for a 
user to learn a new computer language. A discipline oriented 
user is led through the entire sequence of possible processing 
options available by a series of prompts and a layered struc¬ 
ture of menus which require simple responses. 

Typical user data analysis sessions last for about two 
hours. During a session, the user has several options avail¬ 
able to him for displaying intermediate results, generating 
output products, and defining sequences of processing op¬ 
erations. Operations may be terminated with the capability 
to restart in subsequent analysis sessions on the system. 
The user has the option to start and to save/restart analysis 
sessions using magnetic tape and/or disk units. The disk 
operations provide a significant speed advantage over the 
magnetic tape operations. 

The second type of system user, the programmer/analyst, 
typically utilizes operating system facilities and the operat¬ 
ing system commands to code, compile, link, edit, and 
checkout the software being developed. The programmer/ 
analyst does not usually interact in the menu environment 
until the final checkout and integration phase in the devel¬ 
opment of a new applications software package. 

Capabilities are being developed to allow users to define 
and automatically execute a sequence of menu processing 
steps in one interactive operation by specifying a command 
string which is used to initiate and control the desired pro¬ 
cessing sequence. 


HARDWARE DESCRIPTION 

A more detailed version of the AOIPS hardware block 
diagram is presented in Figure 2. 

Figure 2 indicates which devices are interfaced to the 
PDP-11/70 through high-speed direct memory access (DMA) 
channels and which devices are interfaced through the 
UNIBUS input/output channel. The interface between 
IAT2, IAT3, and the PDP-11/70 is a dual interface in which 
all high-volume data transfers use a high-speed channel and 
all low data volume, control signal information transfers use 
a UNIBUS channel interface. 



Figure 2 indicates the functional interconnections of the 
video switching system incorporated into the AOIPS config¬ 
uration. All video signals from the three Image Analysis 
Terminals are routed to the video switching unit. The video 
switch itself is under computer control by the PDP-11/70. 
By utilizing this control link, any video signal within the 
system can be routed to the video disk for storage, to video 
recorders, and to a large Advent TV projection screen or 
other TV monitors interfaced to the switching unit. It is also 
possible to display video which originates in any one of the 
terminals on any other terminal. This capability is utilized 
for applications requiring simultaneous use of multiple image 
analysis terminals during information extraction operations. 
In addition, the video disk unit is used to store sequences 
of up to 600 TV sized (512 lines by 512 picture elements per 
line) images for time-lapsed playback through the system. 

Figure 2 also indicates the 9600-bit-per-second back-to- 
back DL11 teletype interface link between the PDP-11/45 
and 11/70 computers. This link is used primarily to commu¬ 
nicate synchronization and control information between the 
two central processing units during shared disk operations. 

The shared disk unit is used to transfer data between the 
image analysis terminals, to transfer results of a TV input 
scanner operation to Terminal 1, and to provide a medium 
for sharing peripheral equipment between the PDP-11/70 and 
11/45 computers. 

The TV input scanner scans image data into any user 
selected channel of one of the refresh memories of either 
Terminal 2 or 3. The contents of this refresh channel can be 
rerouted to any storage device or any image display unit in 
the system. 
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One of the 88-megabyte disk units shown in Figure 2 is 
dedicated to Terminal 2 users for save/restart operations, 
the second 88-megabyte disk is dedicated to Terminal 3 
users, and the 176-megabyte disk is used to support system 
functions and software development activities. In addition, 
users on either of the PDP-11/70 Image Analysis Terminals 
will be able to select from one to ten refresh memory chan¬ 
nels for their image processing operations. 

The PDP-11/70-S/360/91 computer link shown in Figure 2 
consists of a 4800-bit-per-second remote job entry commu¬ 
nications link. This communications path will be upgraded 
to a 50-kilobit-per-second link in early 1978. It will provide 
capabilities for AOIPS users to perform large computational 
jobs on the high-speed IBM S/360/91 computer and to dis¬ 
play and analyze the results of these computations on the 
AOIPS. The kinds of information extraction tasks requiring 
this support include multispectral classification of large sub¬ 
images or full images and a variety of weather, climate, and 
earth resources modeling activities. 

Experience in interfacing various kinds of equipment to 
the AOIPS PDP-11/70 computer indicates that from 50 per¬ 
cent to 70 percent of the total theoretical system input/output 
bandwidth (5.8 megabytes per second) can be achieved in 
the current configuration. Figure 3 is an estimate of the 
system input/output bandwidth achievable for various levels 
of system activity within the current AOIPS configuration. 

Experience with the AOIPS 88-megabyte disk units indi¬ 
cates that 60 percent to 70 percent of the theoretical input/ 
output bandwidth (675,8400 bytes per second) can be 
achieved in the current system configuration. 

The addition of Terminal 3 resulted in approximately 80 
percent of the achievable system bandwidth being used dur¬ 
ing planned peak load conditions. For this reason, no addi- 
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Figure 3—AOIPS I/O bandwidth vs. system activity 


tional image analysis terminals will be attached to the exist¬ 
ing AOIPS PDP-11/70 computer in the future. 

Further information about the alternative hardware con¬ 
figurations and peak load conditions considered for AOIPS 
is presented in Reference 1. Additional information about 
the existing system capabilities and a complete listing/de¬ 
scription of the system hardware and software is contained 
in Reference 2. 


Image analysis Terminal 1 

The hardware configuration for the AOIPS PDP-11/45 
Terminal 1 subsystem is shown in Figure 4. 

The Image 100 Image Analyzer Console contains special- 
purpose hardware which operates in conjunction with ap¬ 
plications software to rapidly analyze multispectral image 
data. 

The Image 100 console processing hardware is designed 
to process image data, particularly Landsat satellite data, 
and to perform a parallelepiped classification of multispec¬ 
tral data. The user interacts with the Image Analyzer Con¬ 
sole to execute processing functions, to select training sites, 
and to manipulate themes generated as a result of a classi¬ 
fication operation. 

The Image 100 Image Analyzer Console performs the fol¬ 
lowing functions in hardware: 

• Image Correlation—Two images are displayed alter¬ 
nately in a flicker presentation for image comparison 
and registration operations. 

• Image Scaling—Thumbwheel switches are used to per¬ 
form individual amplitude scaling shifts on each refresh 
memory channel. 

• Ratioing—Images stored in the refresh memories may 
be ratioed and the results of the ratio operation dis¬ 
played on the TV display unit. 

• Image Transformation—General purpose transforma¬ 
tion hardware exists to rotate feature axes in spectral 
space. 

• Spectral Signature Acquisition—Hardware exists to 
compile histograms (frequency versus digital gray level) 
of areas specified by the user in each refresh memory 
channel. From these histograms, upper and lower gray 
level limits are obtained to serve as a four-dimensional 
parallelepiped describing the spectral signature of the 
specified area. 

• Cursor Control—A multimode hardware cursor is sized 
and positioned by joystick control for the definition and 
selection of subimage areas. 

• Image Analyzer—Special-purpose digital logic operates 
under software control to store and test for upper and 
lower gray level limits in each refresh memory channel. 
Parallelepiped signature regions are synthesized into 
decision boundaries by the Image Analyzer for classi¬ 
fication operations. 

• Theme Synthesizer—Circuitry is included in the Image 
Analyzer to combine themes using logical AND, OR, 
Exclusive OR, and theme inversion operations. 
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• Color TV Display—A color TV display provides capa¬ 
bilities to display image data, the results of various 
processing functions and thematic results of classifica¬ 
tion operations. Software also exists to generate split 
screen presentations. 

Terminal 1 also incorporates hardware modifications for 
transferring video signals to and from the AOIPS video 
switching unit. 

The two RK05 disk units shown in Figure 4 are used 
primarily for program storage. The dual ported RP04 disk 
shared with the PDP-11/70 is used for data storage and for 
transferring data to and from peripherals attached to the 
PDP-11/70 computer. The back-to-back DL11-C Serial Line 
Interfaces shown in Figure 4 (and in Figure 5) are used by 
the shared disk software to transmit synchronization and 
control information between the PDP-11/45 and 11/70 pro¬ 
cessors. 

Image analysis Terminals 2 and 3 

The hardware configuration for the AOIPS PDP-11/70— 
Terminal 2/3 subsystem is delineated in Figure 5. Terminals 


2 and 3 provide significant image processing capabilities that 
are not available in second generation terminals currently 
being marketed. Figure 1 depicts the terminal analyzer con¬ 
sole as seen by the terminal user. The terminal was designed 
to register, process, display, and analyze digital image data 
for a wide variety of applications. It rapidly performs many 
image manipulation functions in special purpose hardware 
which is set up and controlled by the AOIPS PDP-11/70 
computer. 

Computer control of the terminal hardware allows appli¬ 
cations system designers to include hardware control func¬ 
tions in software designs and provides the capability to uti¬ 
lize the full power of the terminal to meet image processing 
requirements in a highly flexible and responsive system en¬ 
vironment. 

Figure 6 presents a block diagram of Terminal 2 logic 
components. Noteworthy elements in Figure 6 include the 
hardware lookup tables, the digital matrix switches, a light 
pen, the programmable function button array, and the five 
solid state refresh memories. Refresh memory 5 of both 
terminals is randomly addressable to the bit, byte (8 bits) or 
word (16 bits) level. Each of the refresh memories 1 through 
4 are randomly addressable in groups of 16 words. 
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Figure 5—AOIPS PDP-11/70—Terminal 2 configuration 


Terminals 2 and 3 provide hardware capabilities for: 

• Image Display—The contents of each refresh memory 
channel are displayed on individual black and white TV 
monitors. Any refresh channel can also be displayed on 
a high quality, twenty-one inch black and white moni¬ 
tor. False color displays combining any three refresh 
channels as well as pseudo color presentations of indi¬ 
vidual channels are displayed on a nineteen inch color 
TV monitor. 

• Time Lapse Display—The terminals provide time 
lapsed display of up to ten TV sized images in a movie 
loop presentation. 

• Image Correlation—A hardware fade control is used for 
simultaneous display of two images during image reg¬ 
istration and correlation operations. 

• Image Translation—Hardware capability is provided for 
X-Y translation of images within the refresh memories 
under joystick control for rapid image registration. 

• Graphic Overlay—Graphic overlay of image data sets 
with a cursor, a rectangle, randomly shaped polygons, 
plots and data contours is provided. 

• Joystick Control—Computer control of a hardware 
joystick is provided to select the size and position of 
graphic overlays including a multimode hardware cur¬ 


sor, to select single picture elements or entire sub¬ 
images, to control the direction and speed of image 
scrolling operations, to fly one image over another for 
rapid image registration, and to control the speed of 
dynamic color table lookup ‘rolling’ operations. 

• Image Scrolling—Flight line display of image data sets 
is provided under joystick control. 

• Split Screen Displays—Simultaneous multiple image 
split screen displays are provided. 

• Lookup Tables—Hardware lookup tables are used for 
dynamic color rolling for continuous changing of gray 
level—color assignments, radiometric correction, par¬ 
allelepiped classification (up to 9 spectral channels and 
8 classes), level slicing and contrast stretching. Each of 
these operations is performed within V3oth of a second. 

• Light Pen Detection—A light pen is used to outline and 
select irregularly shaped image areas for processing. 

• Signature Acquisition—A series of hardware registers 
is used to rapidly generate gray level histograms of 
image areas for spectral signature acquisition. 

• Theme Synthesis—Thematic information is synthesized 
using the lookup tables and matrix switches. 

• Image Combination—Images in individual refresh chan¬ 
nels can be combined and re-recorded into the refresh 
memory. 
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VIDEO OUTPUT 
> DISC 

RECORDER 7512 


• Scanner Input—A TV input scanner is utilized to digi¬ 
tize information and store it in any refresh memory 
channel. 

• Video Transfers—All video signals can be transferred 
to or from the AOIPS video switching unit and the 
video disk unit. 

• Direct Data Readout—Manipulated images and picture 
elements within a boundary may be transferred back to 
the PDP-11/70 computer for further analysis. 

• Processing Control—A CRT keyboard display unit and 
an array of programmable function buttons are used to 
control the sequence of processing operations and to 
create new analysis scenarios. 

The terminal hardware performs computer controlled 
image manipulation functions in real time while the user 
observes each operation. Image transfers from the computer 
to the terminal refresh memories occur at a rate of two 
images per second from disk units and at five images per 
second from the AOIPS High Density Digital Tape unit. 
Terminals 2 and 3 are fully interconnected to share resources 


thus allowing a user on either terminal to use up to 10 refresh 
memories to support data analysis operations. 

Terminal 2 is used primarily for meteorological applica¬ 
tions in support of severe storms research investigations. 
Terminal 3 is dedicated to information extraction technique 
development in support of severe storm forecasting inves¬ 
tigations utilizing data from multiple satellite sensors and 
related surface truth measurements. 


High Density Digital Tape (HDDT) subsystem 

The HDDT unit is interfaced to the AOIPS PDP-11/70 and 
consists of a Honeywell Model 96 instrumentation tape re¬ 
corder, a Martin-Marietta Data Converter, a Serial Control¬ 
ler Interface (SCI) fabricated by General Electric Company, 
and a DEC Model DR-70 Massbuss Adapter to operate the 
recorder on the AOIPS system. 

The recorder uses one-inch wide magnetic tapes to record 
data on 10 of 14 data tracks. The remaining four tracks are 




























166 


National Computer Conference, 1978 


utilized for time tagging, audio, and two spares. The unit 
operates with data transfer rates of up to 20 megabits per 
second with bit error rates of 1 in 10 6 , and provides for 
storage of approximately 1.4x 10 10 bits per 7200-foot reel of 
tape at a packing density of 20,250 bits per inch per data 
track. 

Six playback speeds are provided, ranging from 120 inches 
per second to 3 3 A inches per second. The unit will be 
operated at IVi inches per second for playback of data into 
the AOIPS yielding an input data rate of approximately 1.5 
megabits per second. A high-speed search capability is avail¬ 
able at 120 inches per second; tape rewind occurs at 200 
inches per second. 

The Serial Controller Interface (SCI) provides the elec¬ 
tronics needed to synchronize the serial input-output bit 
stream. The AOIPS SCI has been modified to subsample or 
average image data to allow selection of a portion of an 
image during data transfers to the AOIPS. This SCI capa¬ 
bility provides options for obtaining subimages and reduced 


images from the full image data sets stored on high-density 
tapes. 


SYSTEM SOFTWARE 

Numerous software packages are available on AOIPS; 
some are geared to the specific requirements of an applica¬ 
tion and some are of a general support nature. Figures 7 and 
8 present an overview of the major software packages avail¬ 
able. 

Both the PDP-11/45 (Terminal 1) and the PDP-11/70 (Ter¬ 
minals 2 and 3) operate under the DEC-supplied RSX-11D 
multiprogramming operating system. All AOIPS applica¬ 
tions packages interface with the user through the previously 
mentioned hierarchical structure of menus. 

Terminal 1 (the Image 100) was delivered with an appli¬ 
cation package (the Image 100 software system) designed 
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Figure 8—AOIPS Terminal 2/3 software overview 


for interactive multi spectral classification of Landsat satel¬ 
lite data. To support the varied requirements of system 
users, several additional AOIPS software packages were 
developed including: 

METPAK —An interactive system which provides image 
navigation, registration and other functions required to sup¬ 
port cloud motion studies, wind vector generation, and wind 
vector analyses. METPAK computes the cloud height and 
the displacement of selected clouds in a series of time lapsed 
digital images. It generates various analyses of the computed 
wind fields and displays TV and hard copy output products 
of images overlayed with plots and data contours of analysis 
parameters. 

CLASSPAK —An interactive maximum-likelihood multi- 
spectral classification package. CLASSPAK accepts up to 24 
channels of input data; provides for statistical analysis of 
the input data; and performs a classification using up to 8 
selected channels. CLASSPAK provides a highly flexible, 
interactive environment for selecting, defining, displaying, 
editing, and combining training sites and training site statis¬ 
tics. CLASSPAK generates classification maps of up to 25 


classes and displays of class correlation and covariance mat¬ 
rices as well as statistical analysis results. 

Aircraft Sensor Analysis Package (ASAP) —An interac¬ 
tive software package for displaying, processing, correcting 
and enhancing multispectral aircraft scanner data. ASAP 
produces hard copy image output products and TV displays 
of processed aircraft imagery. 

AOIPS Support Package (ASP) —An interactive system 
which provides general image display and manipulation ca¬ 
pabilities. ASP performs histogram generation, contrast 
stretching, pseudo color enhancement, and image combi¬ 
nation operations. 

Figures 7 and 8 also list the capabilities of a number of 
other software packages used to manage AOIPS data bases 
and to generate final output products of image analysis re¬ 
sults. 

SYSTEM APPLICATIONS 

During the past year, over 50 applications investigations 
have been supported on AOIPS. These investigations in- 
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Figure 9—Satellite interpretation of the dynamics of severe local storms 


eluded research studies in meteorology, oceanography, hy¬ 
drology, land use, forestry, geology, agriculture and related 
earth resources disciplines. 

Reference 3 details the AOIPS capabilities used in support 
of severe storms investigations, particularly in the area of 
wind field generation and analysis. Additional system ca¬ 
pabilities have been developed to analyze the dynamics of 
rapidly forming storm cells, to generate precipitation esti¬ 
mates from digital imagery, and to provide temperature pro¬ 
files of selected clouds. These capabilities were used to 
produce the applications results illustrated and described in 
Figures 9 and 10. 

The two primary earth resources applications supported 
on AOIPS include image enhancement for identification of 
ground features of interest and multispectral classification. 
Figure 11 describes one hydrological land use classification 
study conducted on AOIPS. 

FUTURE SYSTEM ENHANCEMENTS 

In early 1978, the AOIPS PDP-11/70 will be interfaced 
through a shared RP06 disk unit to another PDP-11/70 pro¬ 
cessor as shown in Figure 12. This second processor will be 


dedicated to deriving atmospheric temperature and humidity 
profiles from GOES-D VISSR Atmospheric Sounder (VAS) 
satellite data as part of the VAS Demonstration project. The 
shared disk link between the AOIPS and VAS ll/70s will 
provide the means of incorporating sounding data into se¬ 
vere storms analyses conducted on AOIPS. 

The figure also illustrates plans for acquiring and pro¬ 
cessing SMS/GOES satellite data in near real time. These 
capabilities will allow investigators to detect and analyze 
severe storms as the phenomena occur in order to develop 
improved storm detection and prediction capabilities for the 
future. 

During the next two years, several other AOIPS enhance¬ 
ments are planned for meteorological and earth resources 
applications including: 

• Enhancing capabilities for monitoring the dynamics of 
cloud formation and severe storm development. 

• Improving precipitation estimation capabilities using 
several estimation techniques. 

• Developing capabilities for multispectral analysis of se¬ 
vere storm data. 

• Applying multispectral classification techniques to se¬ 
vere storm analyses. 
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Processing System (AOiPS). The upper portion of the Western Branch subbasin of the Patuxent River watershed served as a test site for 
classification. The boundary of this subbasin is identified on the 7’4-roinute USGS topographic map. The classification of Umdsat image 1260- 
15201,9 April 1973 and that of landsat image 1350-15192.8 July 1973 each resulted in cover type identification of greater than 90 percent of the 
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Figure 12—AOIPS—VAS system configuration 


• Incorporating a wider range of nonimage surface truth 
and ancillary information into AOIPS information ex¬ 
traction operations. 

• Providing capabilities for registering image and non¬ 
image data from multiple sources. 

• Establishing integrated data bases for storage, compar¬ 
ison and correlation of multisource/multitime image and 
nonimage data. 

An array computer called the Massively Parallel Processor 
(MPP) will also be interfaced to AOIPS in 1981. When com¬ 
pleted, the MPP will utilize an array of 128x128 primitive 
processing elements (i.e., elementary central processing units) 
to perform image processing computations on 16,384 picture 
elements simultaneously. The MPP has been designed to 
operate at speeds of over 5,000 million additions per second 
and over 1,000 million multiplications per second on eight 
bit integers. The system will be fully programmable and 
capable of handling variable word lengths and floating point 
computations. 

An 8x8 breadboard version of the MPP has been suc¬ 
cessfully tested and is scheduled to be interfaced to AOIPS 


in April 1978. Reference 4 provides detailed information on 
the design and functional characteristics of the MPP. 


SUMMARY 

This paper has discussed the configuration and processing 
capabilities of the AOIPS interactive image processing sys¬ 
tem. Unique subsystems for displaying, analyzing, storing 
and manipulating digital image data were presented. Appli¬ 
cations of the AOIPS to research investigations in meteor¬ 
ology and earth resources were highlighted. 

Our experience to date indicates that future progress in 
relating remotely sensed measurements to earth based phe¬ 
nomenon is heavily dependent on an investigators ability to 
interact with his data and to combine multisensor/multitem¬ 
poral data sets in integrated analysis data bases. In most 
cases, this interaction must take place within hours or days 
of the occurrence of an event to preserve the value of the 
information content in the data and to maximize the realiz¬ 
able benefits. 
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THE EARTH RESOURCES SITUATION 

Population growth and demands for higher standards of 
living are creating pressures which affect all aspects of our 
society. The global population, currently near 3900 million, 
is growing at a rate of 75-80 million/year. 

Coupled with population growth, and magnifying the ef¬ 
fect, are rising demands for a more equitable sharing of the 
products and benefits of technology. The products and ben¬ 
efits derive from the consumption of natural resources. As 
societies develop and technological advancement begins, 
individual demands increase resulting in a more rapid rate 
of consumption of resources. Information is necessary to 
enable prudent decisions regarding the management of re¬ 
sources. But understanding alone will not lead to solutions 
on a long-term basis. Knowledge must be current, necessi¬ 
tating continuing inputs of fresh information. The most prac¬ 
tical and economical manner of acquiring repetitive global, 
Earth Resources information is by means of a satellite-based 
remote sensing system. 

REMOTE SENSING DATA SYSTEM 

1. General 

A total remote sensing data system (wherever the com¬ 
ponents are located and whatever the sophistication) com¬ 
prises four distinct operations: 

• Physical data acquisition and preprocessing 

• Archiving for bulk retrieval 

• Production and dissemination of individual products 

• Extraction of the required information from the data 

2. Data Acquisition 

Mechanisms for data acquisition are the same for most 
applications. Global, repetitive, timely, and accurate infor¬ 
mation is a key element. Landsat addresses, and to a con¬ 
siderable extent, can satisfy the requirements for informa¬ 


tion. For this reason, the discussions will concentrate on 
Landsat data, although recognizing the presence of other 
sensor platforms in the NASA program. Not part of the 
system considered here, but often critical to successful so¬ 
lution of a given problem, is the inclusion of data from many 
other sources, such as maps or socioeconomic data. It is the 
task of the data analyst to blend all necessary data, from 
whatever source, to extract the desired information, and to 
present the results to the decision maker in the most opti¬ 
mum form. 

As Landsat data are received by the data collection sta¬ 
tions, they are forwarded to Goddard Space Flight Center 
(GSFC). Until July 1978, GSFC will apply radiometric and 
geometric corrections in making 70mm film masters, which 
are sent to the data archive at the Earth Resources Obser¬ 
vation System (EROS) data center (EDC) in Sioux Falls, 
South Dakota. GSFC can also produce computer compatible 
tapes (CCT) on special order (through EDC) in which radio- 
metric correction has been applied. Beginning in 1978, 
GSFC will deliver digitally corrected digital data to EDC on 
high density bulk tapes as the standard NASA product. 

3. Archiving 

All data are archived at EDC, where they are available 
for reproduction upon request of orders from users. A cus¬ 
tomer service and catalog function is available at EDC to 
assist users in locating required data. 

4. Availability of Landsat data 

Landsat data are currently reproduced and distributed to 
users by three Federal Data Centers operated separately by 
the U.S. Department of Interior’s Geological Survey 
(USGS), the U.S. Department of Agriculture (USDA) and 
the U.S. Department of Commerce’s National Oceanic and 
Atmospheric Administration (NOAA). Data products pro¬ 
vided by the three centers are similar in scale, format and 
type. By mid 1978 a new all-digital system for handling and 
processing Landsat data will be in operation and will be 
providing improved products to data users. A processing 
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system will be implemented whereby the National Aeronau¬ 
tics and Space Administration (NASA) will provide Landsat 
high-density digital tapes (HDTs) to the USGS’s EROS Data 
Center (EDC) which will be the nation’s federal dissemina¬ 
tion facility for original Landsat data products. 

The Landsat high-density digital tapes routinely received 
at EDC from NASA’s Goddard Spacecraft Center will be 
radiometrically restored (for detector gain and offset) and 
geometrically corrected (to a Space Oblique Mercator map 
projection) using a cubic convolution resampling algorithm. 
By means of retrospective user orders, NASA will provide 
EDC Landsat data which are corrected to the Universal 
Transverse Mercator or the Polar Stereographic (over 65° 
latitude only) map projection, resampled with a nearest 
neighbor algorithm, or in an uncorrected high-density tape 
format (i.e., radiometrically restored but not geometrically 
corrected). 

The digital processing system at EDC has the flexibility 
to produce a brightness-frequency histogram showing 
brightness value distribution for each Landsat scene, and 
perform contrast and edge enhancement functions which are 
based on the brightness value distributions. When placing 
an order, a user may select or omit these enhancement 
functions. Film and digital tape products, from the multi- 
spectral scanner (MSS) and the return beam vidicon (RBV) 
camera, will be available from EDC. Negative film working 
masters in a 241mm format will be made on a high resolution 


laser-beam film recorder. Customer products made from the 
film masters will include second generation positive film 
transparencies, third generation negative film transparen¬ 
cies, black-and-white or color composite prints at various 
scales of enlargement, and 70mm film products. Computer 
compatible tapes (CCTs, 800 or 1600 bits per inch) generated 
from high-density digital tapes will include RBV data in 
subscene sequential format, or MSS data in band sequential 
or line interleaved formats. Pixel interleaved format will no 
longer be available. Specified images may be selected and 
specially ordered in an HDT format. 

More information concerning these data products may be 
obtained from: 

User Services 

EROS Data Center 

U.S. Geological Survey 

Sioux Falls, SD 57198 

5. Foreign Ground Stations 

In addition to the data returned to the United States and 
available through EDC, several countries have elected to 
install their own receiving stations. At the present time, 
these are Canada, Brazil, and Italy. In addition, agreements 
have been reached with Iran (station under construction), 
India, Argentina, Zaire, and Chile (the latter two are not yet 
funded). 




NASA and the U.S. climate program—A problem in data 
management 
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INTRODUCTION 

Climate can be thought of as the average of weather. 
Whereas fluctuations in weather on a daily basis are essen¬ 
tially unpredictable, fluctuations in weather over longer time 
scales are part of climate and may well be predictable. The 
goal of the U.S. Climate Program is to help the nation 
respond more effectively to climate induced problems by 
enabling its government to be aware of or to anticipate 
climatic fluctuations and their domestic and international 
impacts and to identify the real and potential impact of man 
on regional and global climate. 1 

In order to deal with the many aspects of climate, NASA 
developed an agency plan, which defines the NASA contri¬ 
bution to the U.S. Climate Plan. In it, NASA chose to divide 
climate into four separate but related activities: 2 

1. Climate A deals with the current state of the climate. 
The goal is to provide timely information in a usable 
form on the current state of the climate and to provide 
for the analysis of this information. 

2. Climate B deals with regional climate on time scales 
longer than a month but shorter than a decade. The 
goal is to predict fluctuations in climate over individual 
regions of several hundred kilometers square. 

3. Climate C refers to climate which occurs on time scales 
of a decade and beyond. The goal is to understand the 
causes of natural changes in the global climate. 

4. Climate X deals with the climate as it is affected by 
mans activities on all temporal and spatial scales. The 
goal is to understand and anticipate the impact of man 
so that appropriate action can be taken. 

Currently, programs related to the analysis and modelling 
of climate are limited and dispersed throughout many agen¬ 
cies in the government. The National Oceanic & Atmos¬ 
pheric Administration (NO A A), The National Science Foun¬ 
dation (NSF), The Department of Defense (DOD), and the 
National Aeronautics & Space Administration (NASA) are 
but some of the agencies and institutions in which research 
into climate is being conducted. However, there is no central 
focus at the present time. No group is responsible for estab¬ 
lishing and maintaining a comprehensive program dealing 


with the many aspects of climate. This situation will change 
with the establishment and implementation of a national 
climate plan which will be integrated not only among do¬ 
mestic agencies but among many nations having related pro¬ 
grams. At present the major agencies involved are devel¬ 
oping, in parallel, the plans for their respective agencies to 
participate in the overall program. NASA’s role in the 
achievement of the objectives of the National Climate Plan 
relates primarily to satellite based instrumentation, the de¬ 
velopment of climate required parameters on a global scale, 
climate modelling and assessment, special studies, and data 
management. 

DATA MANAGEMENT 

The objective of the climate data management system 
is to permit effective use of a global data base for climate 
research. Information contained in the data base must 
allow researchers to answer a variety of questions on 
climate . . . questions related to the impact of climate 
variations on crop yields, energy, land use and water re¬ 
sources; questions relating to the determination and effect 
of gas and particle loading in the atmosphere; questions 
concerning the variability of climate and the relationship to 
the earth’s energy budget, distribution of cloudiness, snow 
cover etc. 

The global data base must facilitate research on a wide 
range of time and space scales. It must be able to aid in the 
development of climate models and provide data for their 
verification. It must support diagnostic studies that investi¬ 
gate the structure of climate changes and facilitate assess¬ 
ment studies as the effect of climate change on food and 
fiber production. 3 

NASA’S ROLE 

Forty (40) parameters have been identified as key varia¬ 
bles requiring observation from space and having a direct 
effect on climate (Table I). NASA’s contribution to the total 
climate data base will be to produce climate data sets from 
its experimental space observing systems and to maximize 
the value of this data for climate analysis and prediction. 
Validated data sets will be provided to NO A A for inclusion 
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TABLE I 


NO. Parameter 

Weather Variables 

1. Temperature Profile 

2. Surface Pressure 

3. Wind Velocity 

4. Sea Surface Temperature 

5. Humidity 

6. Precipitation 

7. Cloud Cover 

8. Boundary Layer Stability 
Ocean Parameters 

4a Sea Surface Temperature 

9. Evaporation 

10. Surface Sensible Heat Flux 

11. Wind Stress 

12. Sea Surface Elevation 

13. Upper Ocean Heat Storage 

14. Temperature Profile 

15. Velocity Profile 
Radiation Budget 

7a Clouds (Effect on Radiation) 

16. Regional Net Radiation Components 

17. Equator-Pole Gradient 

18. Surface Albedo 

19. Surface Radiation Budget 

20. Solar Constant 

21. Solar Ultraviolet Flux 
Land, Hydrology, and Vegetation 

6a Precipitation 

18a Surface Albedo 

22. Surface Soil Moisture 

23. Soil Moisture (Root Zone) 

24. Vegetation Cover (Non-Forest) 

25. Evapotranspiration 

26. Plant Water Stress 
Cryosphere Parameters 

27. Sea Ice (Percent Open Water) 

28. Snow (Percent Coverage) 

29. Snow (Water Content) 

30. Ice Sheet Surface Elevation 

31. Ice Sheet Horizontal Velocity 

32. Ice Sheet Boundary 

Atmospheric Composition 
21a Solar Ultraviolet Flux 

33. Stratospheric Aerosol Optical Depth 

34. Tropospheric Aerosol Optical Depth 

35. Ozone 

36. Stratospheric H 2 0 

37. N 2 0,NO x 

38. C0 2 

39. CFM’s 

40. CH 4 

in their overall diagnostic data base. NOAA will, in turn, 
provide NASA similar data sets from both conventional data 
sources and operational space observing systems. Standard 
weather parameters such as surface temperature, surface 
pressure, temperature profiles, wind velocity and humidity 
will be acquired through the operational observations of the 
World Weather Watch (WWW) Program and will be stored 
and accessed through NOAA’s Environmental Data Ser¬ 
vices (EDS). NASA and NOAA space observation data has 
been reviewed and a decision made that the starting point 
for a climate data base would be January 1973 which coin¬ 
cides with the launches and data availability from the NASA 
Nimbus 5 and the NOAA-1 satellites. The sources of data 
from existing and planned satellite programs required for 


Climate B are summarized in Table II. Additional sources 
for Climate C and X are given in Table III. It is unlikely 
that full extraction of climate parameters can be afforded 
from all sources. As an example, the Scanning Radiometer 
on the NOAA satellites is the most consistent source for 
global cloud data at the present time. Cloud information 
could also be obtained from the Nimbus and Landsat sat¬ 
ellites but the data coverage is insufficient. Another potential 
source for a cloud data set is the geosynchronous SMS 
(Synchronous Meteorological Satellite) and GOES (Geo¬ 
stationary Operational Environmental Satellites). When all 
geosynchronous spacecraft are in place only the polar re¬ 
gions will not be covered. It may be feasible at that time to 
extract only the polar regions from polar orbiting spacecraft. 


DATA BASE 

The information contained in the data base will be acces¬ 
sible at four different levels of processing. Level O is raw 
data as telemetered from the spacecraft. Level I consists of 
data which has been calibrated into engineering units, e.g., 
radiances, microwave brightness temperatures, and located 
with respect to time, orbit, and attitude. Level II refers to 
climate parameters, e.g., sea surface temperature, percent 
sea ice, soil moisture, at full spatial and temporal resolution. 


1. ACR 

2. AVHRR 

3. CCRBE 

4. ESMR 

5. ERB 

6. HCMR 

7. LBMR 

8. LIMS 

9. LRIR 

10. SAGE 

11. SAMS 

12. SAMII 

13. SBUV 

14. SMMR 

15. SR 

16. VISSR 


Acronyms 

—Active Cavity Radiometer 
—Advanced Very High Resolution Radiometer 
—Cloud Cover Radiation Budget Experiment 
—Electrically Scanning Microwave Radiometer 
—Earth Radiation Budget 
—Heat Capacity Mapping Radiometer 
—L-Band Soil Moisture Radiometer 
—Limb Infrared Monitor of the Stratosphere 
—Limb Radiance Inversion Radiometer 
—Stratospheric Aerosol & Gas Experiment 
—Stratospheric & Mesospheric Sounder 
—Stratospheric Aerosol Measurement 
—Solar Backscatter Ultraviolet instrument 
—Scanning Microwave Multichannel Radiometer 
—Scanning Radiometer 
—Visible & Infrared Spin Scan Radiometer 


Level III consists of climate parameters which have been 
spatially and temporally averaged. 


Functionally, the climate data management system must: 

1. Acquire raw telemetry data from the applicable 
NASA sensors. 

2. Catalog, store, and retrieve all information archived 
in the NASA data base at any level. 

3. Calibrate, locate, and extract climate related param¬ 
eters from the raw data. 

4. Facilitate the validation and analysis of information. 

5. Manage and give easy user access to a complete, 
current NASA data directory. 

6. Grid, register, spatially and temporally average cli¬ 
mate parameters. 
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VEGETATION 


SOIL MOISTURE 
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REGIONAL 

RADIATION 

BUDGET 


CLOUDS 



II II II 
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I 
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LEGENO 


■■■■■■ EXISTING DATA SETS 

V/7V 7 771 DATA SETS EXPECTED TO BE ACQUIRED FROM EXISTING INSTRUMENTS 

I-, DATA SETS EXPECTED TO BE ACQUIRED BY NEW INSTRUMENTS THAT 

1 -- 1 HAVE NOT YET BEEN LAUNCHED 

N-5,—6,—G = NIMBUS 5,6,AND G RE5PECT!VELY-T-N = TIROS-N.SMM-SOLAR MAXIMUM MISSION 
S-A= SEASAT-A;AEM=APPLICATIONS EXPLORER MISSION 


7. Provide information to users in pre-selected formats 
or as experimental data sets in non-standard form. 

8. Provide a central interface with NO A A for access to 
and exchange of climate data sets outside of NASA. 

9. Provide a centralized access to climate data sets in a 
manner which is functionally transparent to the user. 

10. Provide acceptable response time to the user. 


Figure 1 presents a functional overview of the NASA 
climate data management system. NASA Goddard will be 
responsible for receiving all raw spacecraft data from NASA 
missions and maintaining a central end to end data directory. 
The functions associated with the transformation of the raw 
data into the Level I data base is the responsibility of the 
NASA center managing the particular payload. As an ex- 
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ample the Earth Radiation Budget (ERB) Satellite is man¬ 
aged by Langley Research Center. Langley, therefore, 
would be responsible for the Level I processing and archival 
functions. The processing of information into individual cli¬ 
mate parameters, i.e., Level II, is the responsibility of the 
NASA Center charged with development of the particular 
climate parameter. As an example, although Langley may 


have the responsibility of processing the ERB data into the 
Level I data base, another NASA Center, Goddard for ex¬ 
ample, may have the responsibility of processing the Level 
I data into a measurement of the solar constant which is a 
Level II climate parameter. Level II data, therefore, will be 
a distributed data base. Each Center has the option to main¬ 
tain its own local data base for the Level II climate param- 


OZONE 


STRATOSPHERIC 
WATER VAPOR 

STRATOSPHERIC 
AEROSOL 
OPTICAL OEPTH 

TROPOSPHERIC 
AEROSOL 
OPTICAL OEPTH 


TABLE III—Basic Additional Data Sources for Climate C&X 
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eter for which it has the responsibility of developing, or 
submit the data to the central data base maintained at God¬ 
dard. If the former course is chosen, the local data base 
must be accessible by the Goddard data management sys¬ 
tem. Regardless of where the Level II data is archived, all 
information concerning its availability will be maintained in 
the central data directory at Goddard. Level III data will be 
centrally archived and distributed by Goddard. In addition 
to maintaining the NASA data base the data management 
system must be able to interrogate, access, and retrieve 
information from the NO A A data base in order to integrate 
data sets in terms of temporal and spatial resolution. 

SUMMARY 

The Climate Program is not short term, but is anticipated to 
extend over decades. To catalog and manage global data in 


all its forms, to facilitate retrieval of large integrated and 
distributed data sets upon user demand, simply to store, and 
access the equivalent of 10 5 digital data tapes, are typical 
problems illustrative of the scope and magnitude of Climate 
data management. It will be the largest, most complex data 
management system ever developed by NASA. But the 
analysis of climate is complex and requires such a system 
if it is going to be effective, and effective it must be if man 
is to benefit. 
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What good is all this data if we can’t use it? 


by GEORGE J. McMURTRY 

The Pennsylvania State University 
University Park, Pennsylvania 


One of the most disappointing aspects of the Landsat pro¬ 
gram to date has been the relatively slow acceptance and 
use of this remote sensing data by federal, state, and local 
agencies. In order to attract a greater number of users of 
remotely sensed multispectral data, several conditions must 
be satisfied. The user must be convinced that he has the 
resources and capability of acquiring, processing, and inter¬ 
preting the data and that information extracted from that 
data will be helpful to him in satisfying the requirements of 
his particular job. The final processed results must provide 
the desired information in an understandable and attractive 
format. Furthermore, the user must be able to obtain the 
final result more cheaply, quickly, and/or easily than by 
conventional methods. Finally, he must be able to integrate 
the data and/or the processed results with other pertinent 
data. 

In the session on Image Processing System Design, two 
interactive systems have been described. The word “inter¬ 
active” itself implies an intimate involvement of the user. 
This type of analysis tends to require a more technically 
sophisticated user. This in turn implies that user agencies 
will require highly skilled and trained personnel who can 
interact with the appropriate processing system. Even so, 
there will still be a need to have interactive systems with 
which a user may interact “in English” using menus, etc., 

i.e., an input-output format which makes the system appear 
to be not only accessible but attractive and easy to use. If 
these conditions are satisfied the user must then decide 
which specific set of hardware and software he should pur¬ 
chase for installation in his own facility. Another alternative 
is for the user to purchase time and perhaps services at 
some other facility. 

In addition to the papers on interactive analysis systems, 
we have heard a description of the design of a large pipeline 
system for handling large quantities of data routinely and 
quickly. The emphasis in such a system is to process a large 
volume of image data with a high through-put for distribution 
to a wide community of users. Such a system has additional 
requirements for reliability, special processing, and accu¬ 
racy. It must take into account the needs and desires of a 
wide variety of users. In determining the types of processing 
available in such systems several questions are pertinent. Is 
it cost effective for NASA to provide the special processing 
required by the user? Or would it be better for the user to 


obtain raw data and to process that data according to his 
own needs? To answer these questions one must determine 
whether it is reasonable to expect the user to have the 
capability to perform such processing. Many users will not. 
This may suggest that if the principal objective is to increase 
the number of users, then any pipeline system should be 
designed for the needs of many relatively small users who 
would not be expected to have sophisticated processing 
capability rather than to design the system for the user with 
the more highly sophisticated equipment and software, even 
though the latter user may be the recipient of a much larger 
volume of data. In general, the user of whom we are speak¬ 
ing will need a product that has been both geometrically and 
radiometrically corrected and referenced to a common co¬ 
ordinate system. As the user community grows, the users’ 
needs and capabilities will change and any pipeline system 
must be flexible and be adaptable to such changes. 

The authors of the two papers on interactive systems have 
stated that there is a need “to combine multisensor/multi¬ 
temporal data sets in integrated analysis data bases” and for 
“registration of the existing geocoded data bases with Land- 
sat imagery ... if the Landsat data is to be truly useful to 
the user community.” This implies that the output of a 
pipeline system should consist of corrected data, fully proc¬ 
essed to a standard geodetically defined grid and it has been 
suggested that this data should be specifically designed to 
overlay groups of 7.5 minute topographical quadrangle 
sheets (1:24,000). Many different projections have been rec¬ 
ommended including SOM, Lambert Conformal Conic, Pol¬ 
yconic, state plane coordinates, equal area, and lattitude/ 
longitude. Taking into account user needs, conformality, 
zone boundary problems, and ease of mechanization, a con¬ 
formal projection such as UTM should be used. The header 
format on data provided to users should include the follow¬ 
ing kinds of information: 

1. Geometric 

2. Radiometric 

3. Historical 

4. Statistical, including reliability and accuracy estimates. 

The format should provide a structure which will handle 
ancillary information in the form of either cellular data, 
numerical lists, or alphameric lists. Interleaving of bands by 
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pixels or by lines should be determined by the needs of the 
users with less sophisticated facilities. If the data demand 
is sufficient, both types of interleaving could be available as 
standard products. Most non-research users will probably 
not be concerned about the types of sampling techniques 
used in correcting the data but they should be informed 
about such techniques and their possible consequences. The 
more sophisticated user may wish to acquire the raw data 
and perform his own sampling and correction for purposes 
of incorporating the data with a particular geobase infor¬ 
mation system. Since inventory, monitoring, and change 
detection are expected to be among the principal applica¬ 
tions of Landsat data by user agencies, it is important that 
the registration of this data with other data sets, or images, 
be well within one pixel. In addition, the user should be able 
to order small segments of data rather than a large standard 
frame. For example, many users work with areas corre¬ 
sponding to single or a few 7.5 minute quad sheets and much 
local processing time could be saved if they could obtain 
Landsat data covering only the area of their interest. To 
facilitate this type of archive retrieval it has been suggested 


that full geometric and radiometric calibration be performed 
by swath. 

There are many other data features and processing con¬ 
siderations which could make the Landsat data more attrac¬ 
tive and accessible to the potential user community. It would 
be extremely helpful if users were able to identify and have 
access to persons or facilitites in their local region who/ 
which could provide the user with information and assist¬ 
ance in obtaining and using Landsat data. Many potential 
users today have the need for, and the interest in using, 
Landsat data but are limited in their hardware and software 
capabilities. There exists a need to provide modified ver¬ 
sions of existing software packages which would fit on the 
smaller computers often available to these users. One of the 
most pressing needs today is the availability of a relatively 
inexpensive method of producing a permanent and attractive 
display of a processed final product. For most user agencies 
a black and white character printout is not an adequate final 
product. The user frequently desires a product in a color 
film format which is inexpensive and which preferably can 
be obtained locally. 
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Electronic fund transfer systems 


Electronic funds transfer systems (EFTS) were considered for several years to 
present significant cost savings opportunities to financial institutions. Ideally, 
their wide scale introduction and acceptance would cut down paper work and 
enhance the market shares of those participating. Retailers offering services 
provided by financial institutions would also benefit directly by reduced check 
costs and possibly by attracting new customers through the convenience of 
banking facilities at their locations. 

For several years, reports of varied systems designs, their promises for ac¬ 
ceptance and the favorable impact on their sponsors were widely circulated. 
However, more recently the pattern of comments has become less optimistic. 
Now several key areas of concern are emerging: 

• What are the roles of financial institutions and retailers as EFT alters the 
types of locations and of services which they can provide? 

• What is the role of government in providing services, particularly when the 
services might not be a natural monopoly? 

• What are the technological impacts of systems which allow for a high degree 
of concentration of information regarding an individual and his transactions. 

• What legal modifications must be enacted due to the consumerist, anti-trust, 
and institutional changes which are brought about by EFT? 

The last few years have demonstrated that the design and development of 
systems to achieve EFT’s objectives may be the easiest part of EFT. Complexity 
arises because individuals and institutions are less willing to accept change than 
had originally been contemplated. Institutions often find more significant disrup¬ 
tions to their business activities than they had expected. Technological change 
is easier to design than to implement. 

The National Commission on EFT, organized to review the societal ramifica¬ 
tions of EFT, did not produce a report which could be easily implemented 
legislatively. Instead, there is still a significant polarization of interests coupled 
with diminished excitement about projects—most less successful than anyone 
had anticipated. 
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However, long term expectations are that certain phases of EFT still have 
significant promise: 

• Automatic teller machine costs are declining rapidly and with more effective 
marketing, their installation will be more widespread. 

• The automated clearing house had to achieve a stage of national interchange 
before it could truly have an impact on corporations. That has now been 
achieved. 

• Designs of systems used at the point of sale still search for the formula which 
provides financial feasibility. Success here will be more difficult to achieve. 

Meanwhile, financial institutions will explore other techniques, both techno¬ 
logical, such as image retention and operational such as truncation, to reduce the 
costs of offering services. Retailers will make more extensive use of their internal 
point of sale systems for more economic transaction processing. 




Systems auditability and control 
in an EFTS environment 


by RUSSELL DEWEY 

SRI International 
Menlo Park, California 


INTRODUCTION 

Losses from accidental and intentional acts involving com¬ 
puters and data communications in financial institutions are 
growing. The current estimate of losses from credit card 
fraud alone is $500 million and could rise to $6 billion to $10 
billion by 1986. Of greater concern than this problem is the 
growing potential for single instances of massive losses as 
EFTS grows and participants become highly dependent on 
continuously available computer services where most of 
their assets are stored in electronic form. 

There is obviously a need to apply the principles of secure 
systems to the emerging development of EFTS, and much 
work has already been completed and published (See Ref¬ 
erences 1 through 9). Most of this work has been focused 
primarily on the technology of security, such as: Personal 
Identification Numbers, Plastic Card Security, Cryptogra¬ 
phy, Physical Security, and Operating System Integrity. 

On the other hand, there is also a growing awareness that 
EFTS security is a “people” problem. The risks, threats, 
and vulnerabilities to EDP systems derive primarily from 
the activities of individuals, either accidental or deliberate. 7,10 
In the EFTS environment, individuals may be employees of 
the financial institutions, merchants, telephone company, 
hardware and software vendors, maintenance personnel, 
computer service bureaus, security personnel, and auditors. 

Between the two orientations—technology and occupa¬ 
tion—a body of information is emerging regarding audit 
techniques, and system application controls that, if imple¬ 
mented, could help reduce the potential for significant loss. 5 
The purpose of this paper is to review and summarize the 
state of that art within the context of the EFTS environment. 


THE EFTS ENVIRONMENT 

The number of financial terminals being installed in re¬ 
mote locations to automate all or part of the transfer of 
credits and debits is increasing. By 1980 there may be over 
100,000 terminals providing a variety of EFTS services, 


including: 

• Deposits 

• Withdrawls 

• Transfers of balances between accounts 

• Direct Debits for purchases 

• Balance inquiries 

• Check authorization and guarantee 

• Credit card authorization and data capture 

• Corporate cash management 

• Funds concentration 

• Corporate to corporate wire transfers 

During the last few years, components of EFTS technol¬ 
ogy have begun to be implemented in a variety of different 
configurations, including shared access networks. A shared 
access network is one which allows for the switching of 
EFTS transactions to more than one possible destination, 
regardless of ownership considerations. Since this environ¬ 
ment is more complex than single institutions dedicated to 
EFTS, we will focus on shared access networks for this 
paper. 

The likely evolution of such shared access networks may 
be described by its principal components: 

1. Remote Terminal—The terminal may be operated en¬ 
tirely by one person, such as an Automated Teller 
Machine (ATM), check guarantee terminal or cash 
management terminal. They may also be operated by 
an intermediary, such as a financial institution teller or 
merchant sales person. 

2. Communications—The terminal may be connected to 
a computer located at a merchant, corporation, or po¬ 
tentially a government facility. In this case, the origi¬ 
nating computer must be connected to the destination 
computer through intermediate computers and tele¬ 
phone communications facilities. Depending on the 
complexity of the local environment, the number of 
institutions participating, and economic considera¬ 
tions, the terminal may optionally be connected di¬ 
rectly to a local financial institution, directly to a joint 
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venture shared EFTS switch, or through common car¬ 
rier facilities directly to the destination. 

3. Files—EFTS files take several forms and may be found 
in several discrete locations: 

• At the remote operator (merchant, corporation, gov¬ 
ernment agency) there may be audit trails of EFTS 
transactions passing through, or decision parameters 
to control the processing of transactions when the 
remainder of the network is down. There are likely 
no balances, account data, or financial institution 
programs. 

• At the acquiring financial institution there may also 
be audit trails, but also balances and account data 
for its merchants and corporate customers and for 
transactions that do not need to be switched, such 
as an “on-us” debit. The acquiring financial insti¬ 
tution may also have decision parameters to control 
the processing of “not-on-us” transactions when the 
remainder of the network is down. 

• At the EFTS switch there may be audit trails and 
decision parameters for any destination facility that 
may be down. There are likely no financial files, 
other than reconciliation totals and settlement 
amounts between institutions. 

• At the destination financial institution there may be 
audit trails, memo-post master file balances, trans¬ 
action files with backup and off-line master files with 
back-up. 

4. Communication Equipment—Each computer site will 
have specialized hardware to interface the EDP system 
to the external communication lines. 

5. EFTS Software (Program Logic)—Each computer that 
is expected to participate in such an EFTS network 
must have specialized programs developed. These in¬ 
clude: 

• Terminal protocols 

• Message format conversions 

• Switching and routing logic 

• Interface logic/protocol to other computers 

• Interface to existing financial software, such as: 

— Demand deposit accounting 
— Savings account accounting 
— Customer information data bases 
— New account processing 

• Specialized audit techniques and application controls 

EFTS APPLICATION AUDIT TOOLS AND 

TECHNIQUES 

The transition from traditional financial institution record¬ 
keeping to today’s EFTS application systems—character¬ 
ized by large on-line master files, lack of manual interven¬ 
tion, and remote terminal entry—has brought with it design 
concepts that cannot rely on traditional manual control pro¬ 


cedures. The accuracy and reliability of EFTS application 
processing is becoming more dependent on the incorporation 
of automated application controls (discussed in the next 
section). The purpose of the EFTS audit tools and tech¬ 
niques is to evaluate those controls, verify processing ac¬ 
curacy and continued compliance with processing proce¬ 
dures. 

Some of the key audit tools and techniques that apply to 
the EFTS environment and verify the correctness of pro¬ 
cessing logic and controls include the following: 

• Base Case System Evaluation —Execute application 
programs against test transaction data, entered through 
a “test mode” EFTS terminal, and compare results 
against pre-determined test results. 

• Parallel operations —Execute new or revised applica¬ 
tion programs and existing application programs and 
compare results. 

• Integrated Test Facility (ITF) —Enter test transaction 
data through live terminals commingled with live trans¬ 
action data. 

• Parallel Simulation —Live transactions are copied and 
processed against auditor programs that simulate pro¬ 
cessing logic. The simulation results are then compared 
to the live results. 

• Transaction Selection —Systematically screening and 
selecting transaction samples entered through EFTS 
terminals for subsequent manual verification. 

• Embedded Audit Data Collection (Sometimes known as 
System Control Audit Review File: SCARF) —Audit 
subroutines are embedded in the application programs 
to screen and select internal generated messages be¬ 
tween EFTS logic modules that result from the original 
terminal transaction. 

• Terminal Audit Software —Direct access to inspect live 
files during actual operation of the EFTS. The auditor 
will want to examine terminal polling lists, routing ta¬ 
bles, merchant codes, floor limits, and default authori¬ 
zation tables. 

• Snapshot and Trace —Secure documentary evidence of 
logic paths, control conditions, and processing se¬ 
quence of a specific transaction. This is done by con¬ 
tinuously recording transaction status for some selected 
transaction as it passes through the system. 

• Job Accounting Data Analysis —Select, extract, and 
display job accounting data to monitor access to sen¬ 
sitive data files and on-line libraries. 

• Code Comparison —Use of an off-line program to com¬ 
pare two versions of an application program to identify 
differences in coding. 

EFTS APPLICATION SYSTEM CONTROLS 

The purpose of the EFTS Application System Controls 
are to assure the accuracy and completeness of the pro¬ 
cessing results, the security of the environment in which the 
EFTS transaction is effected, and the effectiveness of the 
overall computer design and operations. 
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The controls described in the following sections are some 
of those likely to be beneficial in the remote terminal EFTS 
environment. They are grouped according to five phases: 

• Transaction Origination and Data Entry 

• Data Communications 

• Computer Processing 

• Data Storage and Retrieval 

• Output Processing 

TRANSACTION ORIGINATION AND DATA ENTRY 

CONTROLS 

In developing controls within this phase, an organization 
may wish to implement: 

• Special-Purpose Forms —At merchant operated or 
teller operated terminal locations. Proper form design 
encourages completeness and accuracy of data. In the 
cardholder operated terminal environment, Video Dis¬ 
play Units (VDU) may be used for pre-formatted and/ 
or interactive input. 

• Transaction Identification Cross-Reference and Source 
Document Numbers —This control calls for sequentially 
assigned source document numbers (or sequential 
screen numbers) which are transmitted to the EFT pro¬ 
cessor as part of the transaction identification. 

• Sequence Log —An internal log of which sequence 
numbers have been assigned to which merchant/teller 
locations or cardholder operated terminal location. 

• Signatures —In the EFTS environment this control con¬ 
sists of appending a series of endorsements to each 
transaction as an audit trail of each node that handles 
it. It also serves as a “return destination” for undeliv¬ 
erable messages. The first endorsement is the terminal 
I.D. 

• User Identification —A unique identification by class of 
user (cardholder, merchant, teller, bank supervisor) 
used to restrict the transaction to be processed as well 
as files that may be accessed. 

• Batch Serial Numbers —Batches are identified by serial 
number to provide accountability of data, and to assist 
in the isolation of errors when an EFTS terminal cannot 
successfully reconcile at end of day (or other cut-off 
time). 

• Limit the Number of Transactions in a Batch —This is 
done to simplify the reconciliation process. 

• Turn-Around Documents —The source documents con¬ 
tain pre-recorded data in machine readable format. This 
will simplify processing in the event the on-line system 
is down and paper documents are used as back-up. 

• Retention Dates on All Transaction Logs —This is 
based on legal requirements and management policy. 

• Source Documents Maintained at Origin —In the EFTS 
interchange environment this means two things: (1) The 
paper documents (credit card vouchers, debit card 
vouchers, deposit slips, withdrawal slips) are main¬ 
tained as closely as possible to the acquiring bank, to 


prevent unnecessary risk of loss through transportation, 
and (2) the electronic version of the EFTS transaction 
is also maintained at the acquiring facility until balanced 
at the end of the day. Subsequent settlement between 
banks, and actual movement of value data for posting 
can occur in batch mode at lower cost and higher reli¬ 
ability. 

• Error Logging —Maintained by the acquiring facility to 
record and monitor the type of errors occurring at the 
terminal. Suspicious patterns of such errors could imply 
an attempt by an outsider to breech system security. 

• Verification of Re-Entered Data —The data fields on 
resubmitted transactions are subjected to the same ver¬ 
ification procedures as the original transaction. 

• Security of EFTS Terminals —EFTS data entry termi¬ 
nals should be physically secure by placement in a 
lockable room, putting a keylock on the terminal itself, 
or by placing a lockable cover over the terminal device 
when not in use. 

• Terminal Logs —Journals in the terminals to record all 
transactions. 

• Terminal Control Logs —Journals in the terminal con¬ 
trollers to identify imposter terminals on the network 
(i.e. controller total of transactions should equal the 
sum of the legal terminals.) 

• Built-In Terminal and Terminal Controller I.D.’s — 
These devices are provided with electronic identifica¬ 
tion that can be queried by the computer; used to val¬ 
idate proper terminal authorization. 

• Editing and Validating Routines —These may be par¬ 
tially performed in the terminal itself, given anticipated 
hardware capabilities. 

• Passwords —Used to verify that the input is being re¬ 
ceived from an authorized source. Likely to be the 
Personal Identification Number (PIN), although other 
techniques are being actively explored. 

• Unauthorized Access Attempts —An immediate report 
is produced of unauthorized attempts to access the sys¬ 
tem. After a threshold of repeated attempts the system 
shuts down the terminal in question and allows access 
from that terminal only after intervention by security 
personnel. 

DATA COMMUNICATION CONTROLS 

In developing controls within this phase, an organization 
may wish to implement: 

• Secure Phone Equipment Rooms —Locks and alarms 
are used to control access. 

• Network Configuration Polling Table —No open ad¬ 
dresses for unauthorized terminals to gain access. 

• Communication System Control Log —To detect any 
unauthorized changes to the network done through net¬ 
work supervisor terminals. 

• Communication Line Routing —Data communication 
lines are not put through public switchboards (PBX). 

• Local Loop Security —Between the terminal, or termi- 
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nal controller, or computer, and the telephone company 
branch office. 

• Encryption Techniques —Such as the National Bureau 
of Standards (NBS) Algorithm. For an excellent dis¬ 
cussion of encryption techniques, see Kaufman and 
Auerbach. 1 

• Forward Error Correction (FEC) and Automatic Re¬ 
quest for Retransmission (ARQ) —Common types of 
link control; to control transmission errors on each seg¬ 
ment of a transaction path through the EFTS network 
to the cardholder data base and back to the terminal. 

• Message Sequence Number —Each transaction at an 
EFTS terminal may generate several messages (termi- 
nal-to-acquiring CPU, acquiring CPU-to-switch, switch- 
to-cardholder CPU, etc.). This provides a traceable log 
to match inquiry with response, detect lost messages, 
or detect imposter messages. 


COMPUTER PROCESSING CONTROLS 

In developing controls within this phase, an organization 
may wish to implement: 

• Monitoring of Internal-Generated Messages —Internal 
generated messages (such as an authorization request 
to the issuer on an interchange withdrawal transaction) 
should be uniquely identified and cross referenced to 
the external transaction that spawned them. 

• Control Totals —Reasonableness checks and internal 
control totals between program modules. 

• Default Option —Each level in the system hierarchy 
may need to make decisions in default when the rest of 
the system is down, or when it is uneconomical to do 
so (e.g., the merchant bank may authorize a $3.00 trans¬ 
action against a negative file instead of requesting au¬ 
thorization from the issuer). 

• Dual Fields —All entries should be carried as a credit 
against one account and a corresponding debit against 
another, throughout the life of the transaction. 

• Arithmetic Accuracy —Techniques such as double arith¬ 
metic and arithmetic overflow checks are placed in crit¬ 
ical points in the application. 

• Destructive Update —Debit and credit type entries are 
used to correct error conditions, not delete or erase 
commands. 


DATA STORAGE AND RETRIEVAL CONTROLS 

In developing controls within this phase, an organization 
may wish to implement: 

• File Classification —Each file is classified by security 
level, and access is restricted by level. 

• Data Base Control Table —No data base access is al¬ 
lowed unless it comes from an authorized program mod¬ 
ule. 


• Program Linkage Control Table —These tables control 
the authorized module linkage between programs. 

• Dormant Files/Accounts —The system reports on dor¬ 
mant files/accounts that suddenly have activity on 
them. Many attempts at EFTS fraud try to make use of 
dormant accounts. 

• Excessive Activity —The system reports on records and 
data fields that have excess activity over a certain 
threshold. It could mean that a lost or stolen card is 
being used. 


OUTPUT PROCESSING CONTROLS 

In developing controls within this phase, an organization 
may wish to implement: 

• Reconciliation —The response to an EFTS request is 
matched up and reconciled with the original request 
before completing the transaction. 

• Transaction Log —The central transaction log is period¬ 
ically matched against the journal tape in the EFTS 
terminal. These totals are also verified against individ¬ 
ual application control totals. 

• Output Activity Review —Real-time statistics such as 
the number of terminals on-line, transaction quantities, 
and circuit traffic are reviewed by the appropriate man¬ 
agement. System generated subjective judgments, such 
as default authorization, are reviewed by user depart¬ 
ments. 

• Device Verification —EFTS devices whose action such 
as imprinting a card or dispensing cash is controlled by 
the application programs will send a status report back 
to the host computer to verify completion of the re¬ 
quested mechanical operation. 

CONCLUSION 

Any remote terminal-based interactive system will face a 
variety of threats. EFTS, in particular, because of the elec¬ 
tronic movement of funds and value, will be particularly 
vulnerable. Fortunately, a number of system audit tech¬ 
niques and application controls are available to the system 
designer; and more are under development. Depending on 
the particular system design, careful implementation of these 
audit techniques and application controls is likely to di¬ 
minish the vulnerability to those threats. 
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SOCIAL MEANINGS OF ELECTRONIC FUNDS 

TRANSFER SYSTEMS 1 

Electronic funds transfer technologies are rapidly emerg¬ 
ing. The many different technologies, collectively called 
EFTs, provide for the transfer of credits and debits and for 
a variety of related information services. They are likely to 
be widely used and may easily penetrate our everyday lives. 
If they become pervasive, they can alter the texture of our 
society with an intensity similar to that of the radio, auto¬ 
mobile, jet plane, electricity, telephone, and television. 

In their early years, new technologies offer tremendous 
promises to increase the quality of life for their consumers. 
However, when high technologies are adopted on a large 
scale, they often lead to social problems as small problems 
magnify in size. For example, cars were an optional novelty 
and recreational vehicle in the horse and buggy Los Angeles 
of 1900. Today they dominate the social architecture through 
freeways which link sprawling suburban tracts, regional 
shopping centers, and widely scattered workplaces. 

Most complaints about modern “industrial technology” 
focus on large scale, relatively inflexible technologies with 
centrally managed infrastructures. Thus, the automobile, 
television, jet planes, and nuclear power raise troublesome 
problems. On the other hand, individually controllable tech¬ 
nologies such as the telephone, cameras, stereo systems, 
pocket calculators, sewing machines, electric typewriters, 
and photocopiers have been applauded or silently accepted. 
It is not “technology” per se, but the social institutions that 
develop around a technology, that result in praise or mistrust 
and complaint. 

Research studies can shed light on the impacts of different 
EFT arrangements. But investigators make important as¬ 
sumptions when they select particular social patterns and 
values to study. It is easiest to select “dominant social 
trends” such as increasing affluence, urbanization, market 
concentration, and levels of social integration which have 
characterized American life during the postwar years. How¬ 
ever, the criteria for selecting some trends rather than others 
are rarely made explicit. Which trends will characterize our 
futures often lie in the eye of the beholder. 

Trend analyses often obscure the possibility of important 
social choices. If the banking industry has been marked by 


increasing concentration in the last two decades, need that 
continue? Is it necessary or desirable that all EFT arrange¬ 
ments support that “trend”? While most EFT arrangements 
are promoted in a way which is consistent with a consump¬ 
tion-oriented growth economy, are any EFT arrangements 
particularly consistent with a steady-state economy? 2 Such 
questions are hard to ask, let alone answer with casual trend 
analyses. This paper provides an alternative form of analysis 
by explicitly using a theory of social life and a design space 
for EFT arrangements. 

QUALITY OF LIFE 

We experience our social worlds and define the quality of 
our lives in very concrete terms: enjoying coffee with a 
friend, being frustrated over the noise of a radio or plane, 
or being disappointed in love, tense over the prospects of 
losing a job, or amused while watching a child play with a 
cat. “Quality of life” denotes the subjective meanings we 
place on our experiences and opportunities. In general, an¬ 
alysts define a good quality of life both by expressed per¬ 
sonal satisfactions of people and by the absence of discern¬ 
ible social problems such as crime and unemployment. 

Although, there are growing bodies of empirical research 
about perceived quality of life, 3 theories of “quality of life” 
are rare. Gerson, 4 however, has developed a position which 
merits serious attention. It emphasizes the relative power of 
participants in social settings as well as their relative social 
resources and satisfactions. Gerson’s theory forms the basis 
of the analysis presented here. 

We begin with a concept of social order which develops 
in and through a process of ongoing negotiation. 5 In Ger¬ 
son’s perspective, the problem of reconciling the good of 
individuals with the good of the largest social group in which 
they act is that of “creating and managing a pattern of 
negotiations which is viable throughout the range of (social) 
scales at which it takes place.” 4 

This conception of social life emphasizes joint participa¬ 
tion in multiple settings. The subjects of negotiation in each 
setting are flows of resources and constraints upon them. 
Resources include both material goods such as time and 
money, and social goods such as skills and sentiment (pride, 
embarrassment, state of mind, etc.). The quality of life of a 
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participant in a specific setting increases with: 

• the resources at his disposal; 

• his ability to negotiate new resources. 

EFT arrangements may influence the quality of life for 
participants of a social setting on two different levels. First, 
they may influence the resources available to individuals in 
everyday life and the ability of individuals to negotiate with 
each other and with organized groups. Secondly, they may 
alter the resources available to specific organized groups 
and influence their ability to negotiate resources and com¬ 
mitments. In the next section, we will develop six categories 
for describing patterns of resource distribution and negoti¬ 
ating ability. 


CHARACTERIZING QUALITY OF LIFE 

We need a systematic way to think about the influences 
of alternative EFT arrangements upon the quality of social 
and institutional life. We emphasize six characteristics of 
social settings. 

Resource Distribution 

• Opportunity: the kinds of resources that an EFT ar¬ 
rangement adds to a particular social setting. 

• Ecology: The extent to which specific EFT arrange¬ 
ments save or consume disproportionately large 
amounts of scarce non-renewable resources. 

• Coordination: The extent to which specific EFT ar¬ 
rangements alter the ease with which a party can man¬ 
age his activities across all the settings in which he 
participates. 

Negotiating Ability 

• Intelligibility: The extent to which social, organiza¬ 
tional, or technical arrangements can be understood by 
the participants in a specific setting. 

• Dependency: The extent to which the participants in a 
specific setting can choose to use or not use specific 
EFT arrangements without bearing “excessive” costs 
in resources such as money, time, sentiment, etc. 

• Agency : The extent to which participants in a particular 
setting can influence the social, organizational, or tech¬ 
nical arrangements in force. 

In an ideal world, EFT arrangements would increase op¬ 
portunities for all parties, and be ecologically efficient; rel¬ 
ative to current arrangements, they would be more intelli¬ 
gible, increase all party’s sense of agency, and generate so 
little dependency that consumers could choose to get on and 
off with negligible costs. In actuality, these values trade-off 
against each other. 

This paper analyzes the ways in which alternative EFT 
arrangements influence the six aspects of negotiated social 


life. However, first we need to characterize different EFT 
arrangements. 

CHARACTERIZING EFT TECHNOLOGIES 

Most analyses of EFTs are developed in terms of either 
specific EFT arrangements (e.g., point-of-sale networks, au¬ 
tomatic tellers) or specific services (e.g., debit cards, pre¬ 
authorized payments). 6-10 However, for shaping public pol¬ 
icy, it is important to be able to distinguish between EFT 
arrangements (technologies and their associated social or¬ 
ganization) that have different abstract features. 7 For ex¬ 
ample: 

• A point-of-sale (POS) network which serves the major 
department stores to help them improve their cash man¬ 
agement may be more effective if it is national rather 
than regional in scope; 

• A national POS network may be much more convenient 
for people who travel cross country than would a set of 
regional networks; 

• The disruption caused by an unreliable POS network 
would be less severe if it were regional in scale rather 
than national. 

Many discussions of the opportunities offered and prob¬ 
lems posed by different EFT arrangements are reasoned 
through in terms of certain properties of EFTs. For example: 

• Discussions of the accessibility of EFT arrangements 
to smaller banks focus on different ways of sharing 
facilities; 9 

• The impacts of different EFT arrangements on personal 
privacy are discussed in terms of the richness of the 
data they contain and the extent to which people access 
such systems; 11 

Analyzing alternative EFT arrangements can be facilitated 
by developing an appropriate set of attributes to characterize 
them. In this analysis we propose seven “internal” features: 

• versatility : denotes the variety of services provided with 
a particular EFT arrangement. 

• scale: denotes the number of points of contact and 
people served by a particular EFT arrangement. 

• coupling: denotes the extent to which EFTs that pro¬ 
vide separate classes of service or which could be fac¬ 
tored to serve separate geographical regions are joined 
together. 

• reliability: denotes the extent to which specific EFT 
arrangements are trustworthy, e.g., that the equipment 
will not fail and that data or funds will not “leak” 
unexpectedly from the system. 

• richness: denotes the volume and variety of data in a 
specific EFT that can be linked to a particular account 
holder or class of transaction. 

• accessibility: denotes the ease with which people or 
organized groups may gain access to the services or 
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data which are collected or provided by a specific EFT 
arrangement. 

• production costs : denote the costs of fabricating, ad¬ 
ministering, operating, and maintaining a particular 
EFT arrangement. 

These features can characterize a variety of EFT arrange¬ 
ments. For example, under current arrangements, automatic 
teller machines provide a few specialized services, are not 
very versatile, operate on a small scale, are not highly cou¬ 
pled, are relatively reliable, are data poor, and are relatively 
accessible to bank customers. In contrast, national auto¬ 
mated check processing networks would be more versatile, 
operate on a large scale, may or may not be coupled with 
other services, are of uncertain reliability, would be data 
rich and very accessible. 

This set of characteristics can be used to compose many 
important characteristics of EFTs. For example, conve¬ 
nience increases with ease of access, reliability, and versa¬ 
tility. Similarly, Rule’s 11 concept of “surveillance capacity” 
combines our concepts of richness, scale, and accessibility. 
These features are not completely independent. For exam¬ 
ple, the coupling of two diverse EFT operations (such as a 
POS network and a credit-card transmission system) could 
lead to a more versatile system. 

These “internal” features characterize EFTs independ¬ 
ently from the world in which they reside. We also need to 
consider the “fit” between specific EFTs and the social, 
legal, and economic world in which they are embedded. For 
example, automated checking systems will be substantially 
more convenient if electronic records are accepted as proof 
of payment. This motivates our sole “external” character¬ 
istic: 

• fit: an EFT arrangement has a better fit with a given 
social, legal, political, or economic arrangement to the 
extent that the conventions of the two coincide. 


ASSESSING EFT IMPACTS 

The last two sections developed a design space for EFTs 
and a specific set of features to describe the quality of life 
in specific social settings. The analyses developed here will 
be clustered under each of the six aspects of quality of life 
developed earlier: opportunity, intelligibility, dependency, 
agency, ecology, and coordination. Each of these six aspects 
will be considered from the perspectives of (1) everyday life 
of individuals and (2) institutional arrangements. 

Opportunity 

Increases in “opportunity” are the raison d’etre of EFT 
developments. While EFTs may increase opportunities for 
some people, they may diminish opportunities for others. 
Presumably, many individuals would gain access to new 
financial and related services. 10 Yet, some EFT arrange¬ 


ments might endanger the existence of unpopular political 
groups by enabling abusive surveillance and harassment. 6 * 7 

Most EFTs have been developed by private businesses 
which seek to open new markets and increase operating 
efficiencies. And they have been supported by public agen¬ 
cies such as the Federal Reserve Board which seek to en¬ 
hance their institutional efficiency and effectiveness. Since 
most accounts of EFT arrangements emphasize the en¬ 
hanced opportunities the technology provides, they will not 
be elaborated here. 9 * 10 * 12 In general, EFT arrangements will 
offer greater opportunity to larger institutions with increas¬ 
ing levels of scale, coupling, diversity, etc. 

Intelligibility 

The National Commission on Electronic Fund Transfers 
argues that EFT developments should be fostered so as to 
maximize the number of available kinds of EFT-based ser¬ 
vices. 9 Yet, different EFT arrangements can alter the co¬ 
herence of people’s business relations in two primary ways: 

• They can simplify or complicate specific transactions 
and financial contracts; 

• They can alter the demands made upon people to un¬ 
derstand available products. 

As personal choices increase, so do the demands upon 
them for understanding the array of options available and 
the relative benefits of each. We have little understanding 
of the conditions under which people experience increased 
choice as bewildering. 

Some leads may come from research on human memory. 
Twenty years ago, George Miller publicized his famous find¬ 
ing that people could retain “seven plus or minus two” 
chunks of information in short term memory; the number of 
distinct things that people can remember and compare de¬ 
pends upon the ways in which their characteristics are 
coded. It also appears that individuals cannot easily remem¬ 
ber or cope with the details of an arbitrarily large number 
of choices. 13 In economic terms, the “information costs” 
borne by consumers may increase as additional EFT ar¬ 
rangements become available. 

A second issue is the complexity of particular financial 
transactions. Altering the complexity of a few routine trans¬ 
actions, such as paying a utility bill, has minor ramifications. 
In this case, authorized pre-payment schemes may have 
mixed effects: they may simplify routine payments, but 
make it more difficult for consumers to detect and correct 
foul-ups. However, if EFT arrangements become so wide¬ 
spread that they become the medium for most routine pay¬ 
ments for essential services, then individual consumers may 
find the technology not altogether “labor or attention sav¬ 
ing.” 

One societal arrangement is preferable to another if it can 
be easily comprehended by lay people as well as by spe¬ 
cialists. Versatile EFT arrangements which provide a myriad 
of related information services (such as funds transfer, ac¬ 
counting, electronic mail, etc.) to a variety of institutions 
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can easily boggle the minds of most people. Large scale 
EFT arrangements should be judged by a criterion of “man¬ 
ageable complexity.” This means both: 

• that people can understand the contours of an EFT 
system; 

• that EFT arrangements should not confuse the regula¬ 
tory distinctions between banking, communications, 
etc. and the domains of the regulatory agencies such as 
the FCC, FTC, etc. 

EFTs become less coherent as they increase in versatility, 
scale, and coupling. Precisely those features that increase 
the “opportunities” provided by EFTs to some parties can 
render them incomprehensible to the people who are au¬ 
thorized to act on behalf of the public. 

“Intelligibility” is also influenced by purely technical is¬ 
sues. As Weizenbaum 14 points out, very large computer 
programs develop in such a way that they cannot be easily 
understood by their designers or implementors. This issue 
has many difficult aspects. Some concern the limited ways 
we have of representing the design and behavior of computer 
programs. Users and auditors of computer programs have 
occasion to ask very different questions about a set of com¬ 
puter programs and the hardware on which they run: 

• what computations (tasks) can these programs perform? 

• What kinds of computing resources will they consume 
in the course of carrying out a specific computation? 

• Under what conditions will these programs fail to op¬ 
erate properly? 

• Under what conditions will data “leak” from one path 
in these programs to another path in these programs? 

Currently, computer specialists have very few means of 
representing system specifications. These difficulties are 
compounded by the ways in which large software systems 
are developed; on such projects, many architectural details 
which are unspecified in a grand design are resolved (or 
altered) by specialists who are working on complicated sub¬ 
problems. The net effect is that it is common for many 
programmers to introduce minor but significant design de¬ 
tails which are essentially undocumented. 

The quality of available documentation is also strongly 
influenced by the widespread distaste many professional 
programmers feel for documenting their work. While differ¬ 
ent strategies for organizing software production and docu¬ 
ment preparation are being developed to help minimize these 
difficulties, there are, in general, no clear and reliable so¬ 
lutions today. 

Dependency 

To the extent that people depend upon a particular ar¬ 
rangement, they may lose their power to negotiate alterna¬ 
tives later on. Benign technologies provide people discretion 
in their decisions either to use them or drop them should 
they choose to. 


Dependency means both reliance upon a particular good 
or service or reliance upon a single or small set of suppliers. 
While all social life is based on complex patterns of mutual 
dependency, “good” social relations allow some parity be¬ 
tween parties. 

New technologies such as computing, in general, and 
EFTs, in particular, attract many consumers because they 
extend their “reach.” The same can be said for electricity, 
automobiles, and the telephone. Problems emerge when we 
develop complex forms of social organization which are 
dependent upon the reach provided by a single technology 
or system. Thus, the Northeast power failure of 1965 and 
the New York failure of 1977 were particularly disruptive 
because people had few alternative energy sources for illu¬ 
mination and appliances. 

Not all pervasive technologies are essential. Plastic wrap 
could disappear, or even television could be discontinued 
without a major national emergency. However, we would 
have a tougher time cutting back telephones, electricity, and 
automobiles. To the extent that EFT services replace cur¬ 
rent means of payment, they will become more like elec¬ 
tricity than like plastic wrap. 

Particular individuals may organize their lives so that they 
are more or less dependent upon a given technology. Thus, 
in Southern California, some individuals insist on driving 
their cars even one city block, while others make elaborate 
arrangements so they can bicycle to work amidst a maze of 
freeways and boulevards. It is a quite different question 
whether a social collectivity can easily switch from one 
technology to another. 

Payments mechanisms are an essential part of a social 
infrastructure. In that, they are like transportation and per¬ 
sonal communication. Once an attractive new technologi¬ 
cally based capability is developed, many institutions can 
alter their style of business so that they effectively depend 
upon it. 

Large scale, specialized, complex technologies take a long 
time to plan and implement. In the course of developing a 
plan, thousands of people may become involved. Thus, even 
“feasibility plans” develop tremendous inertia. Even when 
major design flaws are found, it is difficult to redesign or 
retrofit large scale systems. The BART system in the San 
Francisco Bay Area provides a particular object lesson 
here. 15 

In general, the larger the scale, versatility, and accessi¬ 
bility and the less costly an EFT service becomes, the more 
likely we are to become dependent upon it. 


Agency 

In the western liberal tradition, personal influence is a 
central moral and political value. In the American political 
system, in theory, the voter is sovereign. And in the neo¬ 
classical theory of the free enterprise system, the consumer 
is held to be sovereign. 16 Social and technical arrangements 
should be valued to the extent that they enhance the sov¬ 
ereignty of individuals. To the extent that they enhance the 
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influence of already powerful groups over weaker groups, 
they should be viewed with caution. 

We have little understanding about the ways in which 
different EFT arrangements alter the influence of people in 
their daily dealings with business enterprises and govern¬ 
ment agencies. EFT technologies can enhance the control 
of either individuals or EFT-using institutions. The actual 
patterns of control that emerge have less to do with technical 
capability than with institutional prerogatives. For example, 
while it is easy to implement “stop payment” mechanisms 
in automated check processing systems, 6 proposals for such 
systems usually neglect these features. In fact, studies of 
consumer preferences seem to show that most people who 
use some combination of cash, checks and credit cards are 
satisfied with the sense of control over their finances that 
these payment media offer. 9 

One may wonder whether particular EFT arrangements 
alter the ability of the polity to act as a collective agent on 
its own behalf. The recent court cases brought by the U.S. 
Department of Justice against IBM indicate that it is easy 
for a corporate giant to overwhelm the resources of a federal 
agency. Specific EFT arrangements may lead to an even 
weaker balance of power between Federal or state regula¬ 
tory agencies and large EFT using institutions. This may 
happen by: 

• certain EFT arrangements fostering further concentra¬ 
tion of capital in selected sectors of the economy. 12,17 
The emergence of “superbanks” would be one such 
development. While there are over 14,000 commercial 
banks in the U.S., over 70 percent of all deposits are 
controlled by the 100 largest banks. 

• EFT arrangements becoming so complex that EFT op¬ 
erations could not be easily influenced by focused col¬ 
lective action such as Federal legislation or regulation. 

These observations don’t lead to easy policy positions 
since federal regulatory agencies have also been an imper¬ 
fect device. They have been criticized for both insensitivity 
to the public, and for being whimsical agents for businesses 
to deal with. 17,18 In general, the larger the scale, the more 
versatile and the more highly coupled EFT operations be¬ 
come, the more they are likely to foster large enterprises on 
one hand, and demands for Byzantine regulatory devices on 
the other. 

Ecology 

In a period of limited resources, we must emphasize tech¬ 
nologies which are energy and resource efficient. Projections 
about energy and resource efficiency and conservation have 
to be carefully assessed for specific technologies in specific 
settings. For example, narrow estimates of the paper-saving 
efficiency of photocopiers would focus on the amount of 
carbon paper saved. However, in most settings where cop¬ 
iers are used, paper and energy consumption has probably 
increased. This is not to say that photocopiers ought to be 
restricted; rather that projections for resource conservation 


need to be made based upon expected volumes of service 
rather than upon efficiency criteria alone. 

On a larger societal scale, the resources consumed by 
particular EFT arrangements must include the costs of cap¬ 
ital construction as well as the resources to operate them. 
In a recent analysis of the economics of mass transit, Lave 19 
has argued that rail systems have lower operating costs than 
automobiles and buses; However, they have higher overall 
costs when one factors in the costs of constructing rails and 
vehicles. Is it possible (or likely) that the same patterns hold 
for large-scale EFT arrangements relative to their manual 
(paper-based) precursors? 

An ironic development with some current EFT arrange¬ 
ments used by banks is that they are used to increase the 
volume of financial transactions generally. 10,20 Thus, a cur¬ 
rent side-effect of some “paperless” EFTs is a net increase 
in the net amount of paper flowing through the banking 
system. 

If energy prices continue to rise, the relative attractive¬ 
ness of EFT based services and its traditional alternatives 
will shift. If particular EFT arrangements are clearly con¬ 
servative of resources, they might be promoted through 
positive public policy. On the other hand, those EFT ar¬ 
rangements that are relatively expensive in energy and non¬ 
renewable resources might be discouraged. 

Coordination 

It is an open question whether specific EFT arrangements 
increase or decrease the ease with which users and others 
can manage their social and business relations. It is a com¬ 
monplace observation that “labor saving” devices seem to 
have exacerbated the plight of the “harried leisure class.” 21 
While EFT advocates proclaim the array of “conveniences” 
that EFT arrangements may provide, these claims seem to 
be oddly isolated from the overall problems that they may 
induce. When they work well, some EFTs may lead to small 
increases in convenience for consumers. In addition, new 
information services, such as accounting-like payment sum¬ 
maries, may help people organize their business affairs. 
However, when payments records are in error, the “costs 
of coordination,” e.g., correcting errors, may increase. Gen¬ 
erally, people can spend less effort coordinating their lives 
in a society that depends upon a large variety of EFT-related 
services to the extent that they lead to greater intelligibility, 
greater agency, and less dependency for people. 

In a recent article, Rose 12 suggests that small and large 
businesses may be able to increase their fiscal control 
through bank-related cash-management services. In theory, 
the same characteristics of EFTs that lead to lower costs of 
coordination for individuals and larger institutions should be 
identical. In practice, the larger institutions can “purchase” 
greater expertise than smaller businesses and individuals. 
Thus, we expect larger institutions to gain more from EFT- 
based information services when they work well. However, 
organizations, large and small, will ease their burdens of 
coordination to the extent that EFTs grow in versatility, 
richness, and reliability. In addition, enterprises with geo- 
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graphically dispersed activities—already a large and growing 
fraction of private enterprise and public agencies—will ben¬ 
efit from increases in the scale of EFT arrangements. 

EFTs and broader social patterns 

This paper has developed a design space for analyzing 
different EFT arrangements. It appears that certain EFT 
designs (e.g., smaller scale, less coupled) present fewer 
potential social problems that EFTs of alternative design. 

Despite these normative considerations, EFT develop¬ 
ments are rapidly emerging in the current industrial system 22 
and the existing legal and regulatory arrangements. Growth 
is a dominant feature of large enterprises; both the computer 
and financial industries are highly concentrated. 23,24 For ex¬ 
pansion seeking suppliers of computing equipment, EFTs 
offer a lucrative new market. For financial institutions, 
EFTs offer strong potential for increasing their market 
share. Firms in both industries may gain from large scale, 
rich, and versatile EFTs. 

As EFT services spread, failing to integrate several related 
services that are manually linked by different users or con¬ 
sumers may appear oddly “inefficient.” Thus, the integra¬ 
tion of large scale, highly versatile EFTs may be less the 
product of a grand design than the by-product of many small 
“locally rational” decisions. Unfortunately, the “tyranny of 
small decisions” easily leads to an overall arrangement 
which no one would have sought had the long run designs 
been made explicit early on. 25 

It is commonplace to notice that some values trade off, 
that others are mutually supportive, and that some design 
values and social values conflict. More reliable technical 
systems often have higher production costs. Larger scale 
and more versatile EFTs which provide more opportunities 
for users and consumers. But they are also likely to be less 
intelligible and diminish agency when compared with less 
expansive EFTs. 

Whether one speaks of value conflicts or “tradeoffs,” the 
social choices we face are no longer very simple. 26 There is 
a cartoon caption that reads: 

There’s a price tag on everything. You want a high stand¬ 
ard of living, you settle for a low quality of life (quoted in 

reference 27). 

In order to help make our choices less grim, I believe that 
we should have clear answers to several questions. These 
are elaborated in the next section. 


RESEARCH AGENDA 

The analyses of this paper indicate some of the social 
dilemmas posed by EFTs. EFTs are not all alike. Some, 
such as automated tellers, may be more amenable to small 
scale operations than, say, automated check processing. 6 To 
the extent that EFTs increase the intelligibility of social 
arrangements, ease of coordination, and social agency, while 


decreasing dependency for the public, they may be thought 
of as especially benign technologies. These hopes may be 
problematic at best and naive at worst. Studies which need 
to be done include those to: 28,29 

• Examine the market structure of firms which use and 
provide EFT services to determine which EFT arrange¬ 
ments foster increased concentration amongst a few 
firms and which arrangements support a diverse array 
of smaller scale businesses. 

• Identify important social costs and develop ways to 
internalize them into pricing schemes for EFTs. Many 
of the social costs of EFTs suggested in this paper and 
elsewhere 6,7 are intangible. Furthermore, they may ap¬ 
pear only after EFTs reach an advanced form of de¬ 
velopment and widespread use. 

• Understand what it means for consumers to have “per¬ 
fect information” about the choice of EFT services. If 
social costs cannot be internalized into EFT pricing, as 
is likely, determine how the public can be best informed 
about what they are. 

• Develop stronger forms of consumer sovereignty in 
oligopolistic markets. It is easy to assume and difficult 
to demonstrate the sovereignty of consumers in com¬ 
plex business and financial markets. The structural role 
of consumers relative to EFT providing institutions 
ought to carefully investigated. 

• Understand which EFT arrangements are most consis¬ 
tent with a low growth rather than high growth econ¬ 
omy. 

• Understand how to develop EFT technologies and their 
associated social arrangements so that they are easy to 
roll-back, if necessary, without major social upset. 

Effective public policies to resolve many of the dilemmas 
raised by pending EFT developments must foster broad 
consensus among social groups with conflicting values and 
interests. In setting a research agenda, we should be fully 
aware that by selecting important values for study, we may 
superficially threaten active interests. This is a problem of 
doing serious research in a highly charged setting. 
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Special purpose terminals 


This technical area focuses on the current state of the art in Special Purpose 
Terminals and provides a forum for discussion of the future trends in Special 
Purpose Terminals. The sessions are balanced to provide sufficient definition, by 
contrasting the Special Purpose Terminal with the General Purpose Terminal, 
along with special aspects of terminal operations such as voice input. 

The criteria associated with planning a special purpose terminal for a given 
marketplace is explored. The major influences exerted by Engineering, Market¬ 
ing, Manufacturing and competition are examined and placed in perspective. It 
gains this perspective by examining basic trade-offs on cost, performance, im¬ 
plementation, marketplace, competitiveness and profitability. 

The areas of market and product existence, product salability, and administra¬ 
tive analysis are examined with details on decision steps required to complete 
the product planning. It highlights the decision criteria and planning process by 
walking through a special purpose terminal that was designed to displace a general 
purpose terminal. 

Making a special purpose terminal work in a general purpose system environ¬ 
ment complements the decision to produce a special purpose terminal. Labor 
costs for factory operation along with the need to reduce inventory costs have 
been a strong motivator for the special terminal in the factory. Most factory 
oriented special purpose job oriented terminals need to directly communicate 
with large scale general purpose data processing systems. This technical area 
presents the system implementation used to connect an interactive factory data 
collection system on line to a large scale IBM data processing system operating 
in an IMS data base environment. 

The factory terminal system has many requirements which include both general 
purpose and special purpose parameters: Time and Attendance (Special Purpose), 
Last Job Checkout (Special Purpose), Inquiry (General Purpose), and Material 
Tracking (General and Special Purpose). 

The method chosen for interfacing the complete system was to emulate a 
general purpose network of IBM 3270 type CRT terminals. The emulation tech¬ 
nique provides for many on-line different modes of operation including the fol¬ 
lowing: 3270 Emulation in the remote communications mode, 3270 Emulation 
with direct channel attachment, Transparent mode of operation, Data collection 
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mode of operation, and Interaction with both main data base and fast response 
local files. 

The future of Special Purpose Terminals is broadly covered with emphasis on 
input devices that complement the environment that we find in the special purpose 
terminal. “The Future of Special Purpose Terminals” examines current trends 
in computer terminal development and extrapolates these into 1978-1988 time 
period, with special emphasis on the transition from large scale integration (LSI) 
to very large-scale integration (VLSI). This session offers projections concerning 
the demands for special purpose terminals and on the ability of the industry to 
respond to these demands. Additional explorations are examined in more detail 
by the panelists who have chosen to concentrate on specific technology areas as 
well as on the extent in which future advances in technology are likely to affect 
terminal systems presently in use. 

One point of view that will be presented asserts that the single most important 
attribute of the interactive computer terminal of the future is that it contains a 
general-purpose computer with the approximate characteristics of a present-day 
mini-computer. Another point of view points out that voice data entry, long 
recognized as an ultimate step towards simplifying communications between a 
human and a machine, is now a reality. Over 200 voice terminals are currently 
in operation in a variety of industrial and commercial applications in eight coun¬ 
tries around the world. New speech input terminals using microcomputers have 
been introduced which can replace and/or complement intelligent CRT/keyboard 
stations by enabling the user to enter data by voice. 

The potential applications of voice data entry are determined by the economic 
advantages of voice input as compared to other alternative data entry devices 
and/or techniques. As the cost of voice data entry terminals decreases, more 
justifiable applications will arise, particularly when the true costs of data capture 
are considered. In some cases, voice input offers advantages which far outweigh 
the direct labor savings and permits certain operations to be achieved which 
could not easily be accomplished using alternative data input techniques. 

A third point of view addresses the impacts of technology advances on current 
data entry terminals and their use. How long does it take for an incumbent 
industry or profession to yield to the onslaught of new advances? It observes 
that keyboard data entry is supposed to have been “dying” for over 20 years, 
ever since the introduction of optical character recognition (OCR). Each time a 
new device is developed the cry “keyboard data entry will become obsolete in 
five years” is heard again. Yet through all of this, there are more data entry 
keyboards of some type in use today, and more are projectd for the future. 




Decision criteria in terminal product planning 

by ERIC C. WESTERFELD 

NCR Data Pathing, Inc. 

Sunnyvale, California 


THE SPECIAL PURPOSE TERMINAL VS. THE 

GENERAL PURPOSE TERMINAL 

The concept of special and general purpose products is, 
to a great extent, in the mind of the end user. The application 
areas have the greatest impact on the definition of special 
purpose and general purpose. Products which address ap¬ 
plications in many areas are considered general purpose. 
Also, products which address no specific application but 
can be adapted with additional software or hardware are 
considered general purpose. 

The concept of the product incorporates the market or 
end usage of the product as special or general purpose also. 
A grid of special purpose and general purpose terminals and 
special purpose and general purpose applications emerges 
(Figure 1). In reality, any attempt to make clear line dis¬ 
tinctions as shown in Figure 1 are not realistic, and the more 
general grid in Figure 2 shows the complexity of this graphic 
determination. For presentation purposes, only Figure 1 
presentation will be used to denote: 

A—A general purpose terminal used in a general purpose 
application. 

B—A general purpose terminal used in a special purpose 
application. 

C—A special purpose terminal used in a general purpose 
application. 

D—A special purpose terminal used in a special purpose 
application. 

At this level, some generalities can be abstracted, and some 
limits to the directions for this paper can be made. 

First of all, the general purpose application with general 
purpose equipment (A) is the realm of the computer center 
or system user, where the manufacturers equipment or OEM 
equipment predominates. As a general application evolves, 
usually additional software is generated, and additional 
equipment is purchased. Special purpose terminals are dif¬ 
ficult to support because of the increased software burden 
compared to manufacturer supplied system interface ori¬ 
ented terminals (Section C). This is an area in which the 
special terminals do not survive well. For the purpose of 
this paper, Section A will not be pursued again. 

Section C, which shows special purpose terminals used in 
general applications, was briefly mentioned above. The spe¬ 


cial purpose terminals do not stand up well in a general 
application market. The major livelihood of this type of 
system results from a terminal addressing a large number of 
general applications, so that its product life can extend be¬ 
yond a single level of application evolution. The identifica¬ 
tion of the general application markets and the development 
decisions will be discussed later in this paper. 

Section B shows a general purpose terminal used in a 
specific purpose application. The typical environment con¬ 
tains one or more general terminals bound by special soft¬ 
ware, often with a dedicated minicomputer or microcom¬ 
puter interfacing a computer system and the special purpose 
application. A heavy burden is placed on terminal support 
software to make the general terminals conform to the spe¬ 
cial requirements of the applications. 

Section D shows the special purpose terminal satisfying 
a special application. The burden on application support 
usually resides in special hardware features or special mi¬ 
crocoding. The trade-offs between Section B and D appli¬ 
cations are the most easily understood. The basic problems 
are availability of components (including major OEM 
pieces), cost of hardware design, cost of software design, 
documentation, maintenance and training. 

The user implementation aspects for these trade-offs and 
decision have been covered in numerous other papers and 
articles. The focus of this paper will be on the manufacturer 
rather than the user. Section B, C and D will be generalized 
as part of a larger problem of product planning in the next 
section, before bringing in examples of the decision making 
for applications. 

THE BASIC INFLUENCES ON DEVELOPMENT OF A 

PRODUCT 

The business units that form the product development 
group are: 

• Administration 

• Marketing 

• Business Development 

• Manufacturing 

• Engineering (Hardware and Software) 

The interaction of these units create the final product price 
and function, regardless of where the initial product idea 
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General Special 


Applications 

Figure 1—Sectional analysis of decision criteria 

originated. Two basic outside influences complete the inter¬ 
actions: 

• Customers/Market Place 

• Competition 

The interaction between all of these groups can be viewed 
as a feedback system using the approaches of Industrial 
Dynamics. Figure 3 shows a simplified feedback system for 
product function and Figure 4 shows a similar function for 
product price. Both price and function are inputs into each 
other’s system interactions. In this simplified format the 
elements for Customer/Market Place and Competition are 
shown as constants but they have their own system dynam¬ 
ics which complicate this picture. For now these simple 



Most Most 

General Special 

APPLICATIONS 

Figure 2—Sectional analysis for real world decisions 



techniques will be sufficient. The major intent is to simulate 
price and function determination in this feedback system 
until it stabilizes. 

As each organization has its own management, depart¬ 
ment dominancies and interactions, there is no single system 
that solves how function and price will be determined. In 
the short run, the system described is likely to be astable. 
The non-linearities and administrative over-rides keep the 
real world stable. The major instabilities in the feedback 
model should be viewed as warnings, to help identify the 
failings and traps that occur in the process of product func¬ 
tion and price identification. 

The major problems of the product idea development are 
documented by many management oriented books and ar¬ 
ticles. Only a summary of problem areas will be covered in 
this paper. 

• Dominance of a business unit—when all areas do not 
interact fully, an unbalance price/function decision may 
be made. Examples are: 

Manufacturing —An easily manufactured product may 
not have features required in the market place. 
Marketing —Features planned for good sales may cause 
poor system performance. 

Engineering —Design techniques may limit product util¬ 
ity or adaptability. 

Business Development —Long range goals may make 
the product initially unsalable. 
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• Inability to play “devil’s advocate”—when a product 
idea is developed and all parties “jump on the band 
wagon,” the balancing effects to round out a product 
may be missing. 

• Communication problems may make all parties agree 
on the words used but have different concepts on the 
product usage, function and marketplace. 

The “tales of woe” can run on indefinitely, but only a 
sample is necessary to realize that product planning is an 
art still rather than a science. The intention here is to skip 
this phase, and proceed to the business decisions in a ter¬ 
minal satisfying Sections B, C and D of the general purpose 
versus special purpose product. 

PRODUCT TRADE-OFFS 

For every user trying to decide which terminal to use and 
how to use it, there is a producer looking at that same 
problem. The producer has already finished his decision 
making, armed with the tools of market analysis and statis¬ 
tics. The producer also has his judgment and best guess 
working for him. When he produces a terminal product, the 
following factors lead to the general purpose versus special 
purpose trade-offs: 

• Function 

• Cost 

• Performance 

• Implementation 

• Marketplace Size 

• Competitiveness 

• Profitability 


These areas of decision and trade-off are interactive, in that 
a change in one will usually affect another. 

An example of the product planning trade-offs can be 
shown for all these factors. 

• Function —Function is the easiest factor to handle. The 
first user decision and equipment selection usually is 
“Does it do what I want?” Functions planned for a 
terminal product must be competitive to existing ter¬ 
minals already marketed. In a “plug for plug” replace¬ 
ment market, the unit must duplicate its competitor, 
and in a general market, the unit must exceed its com¬ 
petitor. This results in the terminal features escalation 
trends in the industry. A side note on function is that, 
if a good size market is buying equipment with “bells 
and whistles” it does not need but has no choice, re¬ 
ducing functions produces a special purpose terminal, 
generally at a reduced cost, and generally with a re¬ 
duced market. Note the interaction with Marketplace 
Size, Cost and Competitiveness. 

• Cost —The hardest factor to pin down is the cost trade¬ 
offs. The second user decision on equipment selection 
usually is “Can I afford this?” If the equipment price 
is out of line with the competition in a marketplace 
segment, the equipment will quickly disappear from the 
market without some over-riding function or perform¬ 
ance enhancement over the competition. Cost goals 
during product planning are made reflecting the product 
nature, split into two basic types: Price Cutters and 
Function Raisers. Price Cutters must always produce 
lower cost products, generally with identical functional 
features. Function Raisers must always produce more 
functional features (including performance), generally 
with a higher cost. Occasionally both can be combined 
and usually appear in the plug to plug replacement mar¬ 
ket. As cost goals are determined, the product is pushed 
into a special purpose or general purpose direction or 
preference. Interactions on cost are Function, Compet¬ 
itiveness and Marketplace. 

• Performance —This is often viewed as a part of func¬ 
tion, but is separate. A 300 Baud terminal and a 110 
Baud terminal may have identical functions, but offer 
distinct performance differences. The tradeoff with spe¬ 
cial purpose and general purpose performance require¬ 
ments are strongly coupled to price. The user wants all 
the performance he can get, if it does not cost him more 
money. Likewise, the user does not want to pay for 
any performance his application does not require. Both 
ends of the performance spectrum lead to a special 
purpose terminal, mainly because it attracts a limited 
marketplace that requires the additional performance, 
or desires the cost benefits. The interactions are Cost, 
Competitiveness and Marketplace. 

• Implementation —Implementation is the most difficult 
portion of the trade-offs to judge. The user sees the 
packaging and styling and interacts with the operational 
characteristics. The external features are generally sec¬ 
ondary to the equipment user. Internally, the product 
is strongly shaped by implementation. Functional and 
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performance growth are effected. Cost and profitability 
are determined by how the equipment can be manufac¬ 
tured and maintained. The producers emphasis on man¬ 
ufacturing or engineering can shape a product’s long¬ 
term market and make it general or special purpose to 
achieve a market share and production volume required 
by the producer. A producer with low volume and R&D 
emphasis will tend toward specialized implementations 
and many product upgrades. A producer desiring high 
volume and low manufacturing costs will tend toward 
a generalized implementation to widen the marketplace. 
Implementation interacts with all the product trade-offs. 

• Marketplace Size —The product direction to a wide or 
narrow market with a small or large volume is a pro¬ 
ducer’s trade-off. The general purpose terminal usually 
is aimed for a large volume with a wide market. The 
special purpose terminals is usually aimed at a narrow 
market with a substantial volume. Some special purpose 
terminals are aimed over a wide market with low vol¬ 
umes in many narrow segments of the market. The 
decision on customer identification and subsequent 
marketplace size trade-offs should occur early in the 
product development cycle to allow the follow-up trade¬ 
offs on Function, Cost, Performance and Implementa¬ 
tion. The marketplace determination is one of the most 
difficult ones, and the equipment user interacts in a 
secondary way, unless equipment is being manufac¬ 
tured to order for the user. The user interacts princi¬ 
pally through surveys, sales representatives, and most 
important, by buying to his preferences in the equip¬ 
ment market. 

• Competitiveness —This goes ‘hand in hand’ with mar¬ 
ketplace size and forces the decisions on Function, Cost 
and Performance. The decision on how to compete and 
with whom to compete is part of all the previously 
discussed areas. This item also impacts all the above 
areas because timing of a product release and its sub¬ 
sequent distribution enter into the development cycle. 
The competitive factor pushes many trade-offs with a 
business decision to pursue markets, and also, how and 
when to do it. This factor in particular makes the sim¬ 
plified Figure 1 general versus specific market into the 
complicated Figure 2. 

• Profitability —Last but not least important is the prof¬ 
itability trade-off. Profitability is almost always as¬ 
sumed, but is a trade-off along with competitiveness. 
Profitability is essential for a producer, but the degree 
and timing of profitability is a trade-off factor. The 
producer may not become profitable without limit, be¬ 
cause the cost will not be competitive or the product 
will not be purchasable. The profitability in the early 
stages of a product may be a trade-off against long-term 
profitability. This factor affects all the other areas be¬ 
fore manufacture and throughout the product life. As 
an unwritten goal it shapes many of the decisions pro¬ 
ducing special purpose or general purpose terminals. 

The product trade-offs that occur in the development 
cycle affect the production of special versus general purpose 


equipment. The next step is the actual producers decisions 
which shape the product and its applications. 

DECISION FACTORS IN PRODUCT PLANNING 

The trade-offs, detailed above, lead into the actual product 
planning decisions. The decisions are the end result of the 
product trade-offs and are the (hopefully) formal documen¬ 
tation of the product direction. In order to concentrate on 
the basic decisions for product development and planning, 
a simple technique of ‘problem busting’ will be used. Prob¬ 
lem busting is the subdivision of a major decision—in this 
case, product development—and generating simpler prob¬ 
lems that may be attacked. In a sense, there will be a re¬ 
combination of the areas discussed in the trade-off section, 
and responsibility will be assigned to the previously dis¬ 
cussed business units. 

The first major problem busting will uncover three basic 
problem areas for a product: 

Existence Decisions 

Salability Decisions 

Administration Decisions 

Each of these areas will be discussed and subdivided further 
to allow proper product analyzation. 

The first area of existence decisions concerns fitting a 
planned product into the real world. The first subdivision is 
the existence of customer need. Does a customer base exist 
that needs the product? Does the customer base have the 
ability to buy the product? The existence of a ready to buy 
market produces products different from the products re¬ 
quiring an education before customer acceptance. The de¬ 
cision on existence of customer need is most influenced by 
Marketing and Business Development. The second subdi¬ 
vision is the existence or product viability. Can the product 
be implemented within the existing or developing technol¬ 
ogy? Can the product be manufactured within the existing 
capabilities? Will the product components be available in 
the future? The decision on the existence of product viability 
is most influenced by Manufacturing, Engineering and Busi¬ 
ness Development. The questions shown here break the 
problems down another level but will not be detailed here. 
The existence decisions should be made first in the product 
planning of equipment. 

The second area of salability decisions concerns the de¬ 
termination of the product characteristics including the char¬ 
acteristics which make the equipment general purpose or 
special purpose. The first subdivision is performance and 
function salability. Can the unit be sold with the planned 
functions? Can it be sold with the planned performance? 
Does the unit still attract the customer base anticipated in 
the section on existence? These are most influenced by 
Engineering, Marketing and Business Development. This 
decision formalizes trade-offs on performance and function. 
The second subdivision is pricing and timing salability. Will 
the product sell to the customer base anticipated? Is the 
product on the market in time to sell or too early to be 
accepted? Can the product distribution handle the product? 
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These are most influenced by Manufacturing, Engineering 
and Marketing. This decision begins to lock the product into 
its role as special or general purpose. The third subdivision 
is salability against competition. The marketplace and the 
existing competition are analyzed against the planned prod¬ 
uct capabilities. Can the unit out-price the competition? Can 
the unit out-perform the competition? Can the unit ‘steal’ 
new business from the competition. Can the unit displace 
installed products of the competition? Will the unit displace 
the company’s own installed products? The major influences 
on this are Marketing and Business Development. At this 
point the product characteristics are firm. The next major 
set of decisions cover the actual go ahead to produce a 
product. 

The third area of administration decisions produces the 
final product with any long range modifications to the 
planned product. The first subdivision is administration of 
the risk and cost to produce. Can any component not be 
available in the future? Can any design problem not be 
solved within fixed constraints of the product? Can key 
individuals leave the project? Can manufacturing make the 
quantity forecast economical? Can any Assembly, Test 
and Quality Assurance areas invalidate basic design areas? 
This decision is influenced by Manufacturing, Engineering 
and Marketing. The second subdivision is administration of 
profitability. Can manufacturing cost and sales price pro¬ 
duce a profit with known overhead? Can cost improvements 
be made? Can manufacturing improvements be made? Can 
component costs drop with volume and time? Can labor 
costs be handled? This decision is influenced by Engineer¬ 
ing, Marketing and Manufacturing. The third subdivision is 
the administration of corporate requirements. Can this unit 
produce subassemblies in common with other equipment? 
Can this unit use subassemblies from other equipment? Can 
this unit solve an outstanding problem for another portion 
of the corporation? Can this unit produce side usage in other 
areas of corporate business with minor changes? This last 
area is influenced by all business areas including Adminis¬ 
tration. At this point the last of the product changes is 
made and the decision to go ahead with a product occurs. 
Function and detail specification and marketing literature 
follow in time, but now the product has become general or 
special purpose as a result of the previous trade-offs and 
decisions. 

The intention of discussing decision criteria has stayed 
away from actual mention of terminals and remained as a 
general analysis. An example of the product planning in a 
terminal design will be shown in the following section. 

EXAMPLE OF A SPECIAL AND GENERAL 

TERMINAL INSTALLATION DECISION 

NCR Data Pathing, Inc. produces factory data collection 
equipment and systems as front ends for MIS oriented large 
computer systems. A special purpose data collection ter¬ 


minal, the 103, offered an employee badge reader, a punched 
card scanner, and keyboard input as well as 16 character 
message display and a backlighted message panel for output. 
This unit received wide usage in a narrow market. 

In many installation sites, MIS capability was brought out 
to the factory floor with an IBM 3270 CRT display with 
keyboard, allowing a general inquiry system to the host 
computer system. These units often physically resided next 
to each other. The IBM 3270 is a very general purpose 
terminal and was used in conjunction with a special purpose 
terminal in a fairly clumsy manner. As a large number of 
these combinations showed up, a special purpose terminal 
with a degree of generality was planned to replace the com¬ 
bination. 

DPI introduced the MIT-133 as an IBM 3270 replacement 
specifically for the factory data collection environment. The 
MIT-133 had the IBM 3270 CRT and keyboard, but also 
added a special keyboard for factory environment with large 
keys and simplified sequential positioning. Also the badge 
reader and card scanner were added. The keyboard was 
placed at an angle to make operation easier from a standing 
position which is typical in factory inquiry, and a quadruple 
size character font was added to improve readability of the 
typical short factory messages. This special size character 
set replaced the 16 character message display, and the back¬ 
lighted message panel. Special software in the DPI front-end 
formats the CRT screen with the same data editing capabil¬ 
ities available on the DPI 103. 

The unit is a special purpose terminal in a special purpose 
application displacing a general purpose terminal in a special 
purpose application. The unit is a function raiser not a price 
cutter, and the MIT 133 is actually higher priced than the 
IBM 3270. This unit has extended functions that get the 
factory data collection job done more efficiently, and as a 
result, save considerable cost from the main job of MIS, 
saving manufacturing money. 

The decision process covered the customer need and 
product viability, demonstrated by the actual co-existence 
of the DPI 103 and the IBM 3270. Performance matched the 
existing units and was improved by creating a single data 
entry terminal unit. Pricing and timing recognized additional 
capability for additional cost when there was a performance 
benefit. Competitive capabilities were satisfied by incorpo¬ 
rating all IBM 3270 features as well as adding new capabil¬ 
ities. Administrative decisions cannot be detailed. 

The decision shown here is only a Section B versus Sec¬ 
tion D, but the decision criteria are valid across all product 
planning areas. 
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Making a special purpose terminal work 
in a general purpose systems environment 


by DONALD J. BIRMINGHAM 

NCR Data Pathing Inc. 

Sunnyvale, California 


THE FACTORY DATA SYSTEM 

Factory Data Systems have many application require¬ 
ments not normally satisfied by standard general purpose 
terminals. In the past twenty years there has been an evo¬ 
lutionary growth of job oriented terminals to satisfy the 
unique requirements of the factory. To maintain the pace of 
the current expanding system requirements, the special pur¬ 
pose terminals must communicate on a transaction basis 
with a general purpose data processing system. 

A schematic of a comprehensive factory data system is 
shown in Figure 1. The factory terminal system has many 
requirements which include both general purpose and spe¬ 
cial purpose application parameters in the following areas: 

• Time and Attendance 

• Labor Distribution 

• Job Tracking 

• Material Control 
—Raw Material 

—Work in Progress 
—Finished Goods 
—Shipping and Receiving 

• Inquiry 

• Tool Tracking 

• QA Reporting 

Most of the above terminal applications must communicate 
directly with a large scale general purpose data processing 
system. This paper outlines the system implementation that 
Data Pathing Inc. uses to connect an interactive factory data 
collection system on line to a large scale IBM data process¬ 
ing system. DPI offers special purpose job oriented termi¬ 
nals, general purpose terminals, and terminals which alter¬ 
nately operate in a special purpose job oriented mode and 
a general purpose mode. 

THE SYSTEM FLOW 

Figure 2 presents the desired flow of transactions from 
the job oriented factory terminals through a DPI system 
control processor to a “host” general purpose data proc¬ 


essing system. Not all transaction types have the same real 
time processing requirements with the host data processing 
system. Some of the different system requirements include: 

• Immediate and Mandatory Interaction 

• Queuing for Load Leveling 

• Queuing for Time Dependent Processing 

• Processing Off-Line for Faster Response 

• Processing Off-Line for Back Up Purposes 

In most systems the main plant data base is located on the 
host processor. A prime example of this in the large IBM 
environment is an IMS data base oriented system. 

TYPICAL FACTORY JOB ORIENTED TERMINAL 

A typical job oriented special purpose factory data collec¬ 
tion terminal is detailed in Figure 3. This particular terminal 
is designed for “community” applications. A community of 
workers runs random or time dependent application trans¬ 
actions on the terminal. Examples of the transactions are as 
follows: 

Time Dependent—Time & Attendance (In & Out) 

—Last Job Checkout (Shift Change) 

Random —Material Move (Function Complete) 

—Material Receipt (Material Available) 
—Labor Transaction (Job Complete) 

The DPI SDT 107 is a multi-function terminal which can 
run the transaction applications gamut from time and at¬ 
tendance through material move to limited inquiry. To do 
these transactions the terminal incorporates the following 
features. 

• Transaction Descriptions (Applications Menu) 

• Transaction Select Keys 

• Operation Instruction Mask 

• Function Indications 

• Keyboard (Numeric and Alpha) 

• Alphanumeric In Line Field Display 

• Function Keys 
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• Badge Reader 

• 80 Column Punched Card Reader 

• 22 Column Punched Card Reader 

THE MULTI FUNCTION FACTORY TERMINAL 

The growth of different application terminal requirements 
has led to the side by side placement of both job oriented 
and general purpose terminals each installed for specific 
applications. As new applications are added it becomes a 
problem as to which terminal the transactions are assigned 
as most applications have both general purpose and special 
purpose attributes. To solve this dilemma DPI developed 
the MIT 133 Factory terminal. This terminal can alternately 
operate as either a special purpose job oriented terminal or 




Figure 4 —MIT 133 display station 


as a general purpose keyboard CRT terminal. In the general 
purpose mode it operates as an equivalent IBM 3277 ter¬ 
minal. As shown in Figure 4, it has both the main attributes 
of the general purpose and the job oriented special purpose 
terminal. 

General Purpose Special Purpose 

15 inch CRT Screen w four different screen sizes 
Alphanumeric keyboard w special job oriented layouts 
Basic system indicators w Expanded System Indicators 

Optional Badge Reader 
Optional Tab Card Reader 

Figure 5 presents a point by point feature comparison 
between the DPI MIT 133 and the IBM 3277. Most of the 
added features were specifically added to allow for alternate 
or mixed operation in the general purpose or special purpose 
job oriented mode. 


INTERFACE CONTROL SYSTEM 

The DPI 3270 Interface Control System (ICS) is an op¬ 
tional system module to interface a DPI system with any 
host processor system that will support the IBM 3270 ter¬ 
minal system. The module is independent of the specific 
access method used in the host computer system. The ap¬ 
plications can be programmed for interaction with IMS, 
CICS, TCAM, TOS or any other access method which sup¬ 
ports the IBM 3270 general purpose CRT terminal system. 

Within this capability the DPI system has a full range of 
unique interface arrangements to enhance the power and 
flexibility of the terminal applications. 

ICS is an emulation of the IBM 3270 information display 
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FEATURES 


DPI DEVICE BEING 

MIT 130 EMULATED 


LIGHT PEN 


AUDIBLE ALARM 

V 

KEY LOCK 

NOT REQUIRED 

MAGNETIC CARD READER 


SYSTEM INDICATORS 

13 

CHARACTER SIZES 

2 

PRINTER SPEED 

120 cps 

PRINTER CABLE 

2 WIRE 

IN-PLANT LINE USE 

2 WIRE 

EMPLOYEE BADGE READERS 

✓ 

CARD READERS 

V 

NON-CLERICAL KEYBOARDS 

V 

BLINK FIELDS 

s/ 

STANDING OPERATION 

V 

PLANT TIME SYNCH 

✓ 

DUAL TERMINATION 

✓ 

DATA COLLECTION 

✓ 

HOSTILE ENVIRONMENT DESIGN 

v/ 


Figure 5—MIT features comparison 


s/ 

V 

4 

1 

66 cps 

COAXIAL 

MODEMS 


system. Two types of system configurations are supported 
within ICS: 

Figure 6: Direct IBM channel attachment referred to a 
local attachment (Emulation of IBM 3272) 
Figure 7: Communications line attachment hereafter re¬ 
ferred to as remote attachment (Emulation of 
IBM 3271) 

The figures show the “Virtual System” which the host IBM 


processor system “thinks” it is controlling and the “Real 
System” which is being implemented by DPI. 


SPECIFIC TERMINAL OPERATION UNDER ICS 

Under ICS DPI terminal can operate in either the “trans¬ 
parent” or “data collection” mode. 

• Transparent Mode is a one for one operation of a DPI 
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VIRTUAL SYSTEM 
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Figure 6—3272 local attachment 
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Figure 7—3271 remote attachment 
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terminal as an equivalent IBM 3277 terminal. The 
screen format control resides in the host processor and 
all transaction processing from an applications stand¬ 
point is executed in the host processor. 

• Data Collection Mode can be either a one for one or 
group operation of DPI terminals for IBM 3277 termi¬ 
nals. The screen format control resides in the DPI pro¬ 
cessor. Final transaction processing can be executed in 
either the DPI processor or the IBM host processor. 

A terminal such as the DPI SDT 107 is normally applied 
only under the data collection mode of operation. As an 
example it is not uncommon for each of two hundred DPI 
107 terminals to appear to an IBM system as the physical 
equivalent of two IBM 3277 terminals (one for input and one 
for output). Limited job function terminals such as a dedi¬ 
cated time and attendance terminals are also included under 
the same emulation mode. The logical terminal addressing 
is accomplished in the host processors applications. 

The DPI MIT 133 terminal can be applied either in the 
“transparent” or “data collection” mode of operation. It is 
not unusual for the MIT 133 terminal to be applied in the 
mixed mode where specific transactions are transparent and 
others are run in the data collection mode. In this type of 
application a specific terminal may appear to the host pro¬ 
cessor as two different logical terminals (one transparent 
and the other a data collection mode terminal). In such a 
case the transparent mode of the multi-mode terminal mode 
would appear to the host as a one for one physical terminal. 
The data collection mode of the same terminal can appear 
as either a separate one-for-one physical terminal or grouped 
as an individual logical terminal with other data collection 
mode terminals under a single physical virtual terminal. 


A data collection mode application can be processed 
against both a reference (or fast reaction) data base in the 
DPI processor, and the main data base in the host processor. 
A “work in process” (WIP) file is the typical factory envi¬ 
ronment example of the type of data base located in the DPI 
processor. 


SUMMARY 

• The concept of having a special purpose terminal sys¬ 
tem appear to a host processor as a general purpose 
terminal subsystem has provided a most flexible system 
method of interfacing factory job oriented terminals to 
general purpose data processing systems. 

• The concept coupled with special purpose terminals 
such as the DPI MIT 133 designed to operate in both 
the general purpose and special purpose job oriented 
mode provides for multi-operation of software control 
executed from either the best system or the DPI pro¬ 
cessor. 

• The IBM 3270 is an ideal emulation target for providing 
multi-function on-line operation of job oriented termi¬ 
nals. 

• Potential users need not worry about obsoleting existing 
working 3270 CRT software in the host processor when 
adding or evolving to a more sophisticated interactive 
on-line factory terminal system. 

• The implementation of virtual logical terminals frees 
the software writer on the host from redefining all com¬ 
munications devices as new applications and terminals 
are added or modified. 



A panel session—Future developments of special purpose 
terminals 


SESSION CHAIRMAN—REIN TURN 

TRW Defense and Space Systems Group 
Redondo Beach, California 


Panel Members 

Carolyn Dunning—PCC Business Systems 
Santa Ana, California 

M. B. Herscher—Threshold Technology Inc. 

Delran, New Jersey 

Robert H. Anderson—The Rand Corporation 

Santa Monica, California 


INTRODUCTION 

Many types of special purpose terminals are being developed 
for various applications to increase the performance and 
cost-effectiveness of man-computer interfaces. As discussed 
by the speaker and panelists in this session, future advances 
in computer technology will provide new design options, 
such as providing built-in intelligence through the use of 
microprocessors and memory chips. Thus, in future termi¬ 
nals it will be possible to do much more processing and 
store much more information than at present, and it will be 
possible to incorporate more effective security features and 
use more natural man-computer communication modes. 

Donald Kovar in his paper “The Future of Special Pur¬ 
pose Terminals” (in this volume) examines current trends 
in computer terminal development and extrapolates these 
into 1978-1988 time period, with special emphasis on the 
transition from large scale integration (LSI) to very large 
scale integration (VLSI). He offers projections concerning 
the demands for special purpose terminals and on the ability 
of the industry to respond to these demands. His explora¬ 
tions are examined in more detail by the panelists who have 
chosen to concentrate on specific technology areas as well 
as on the extent in which future advances in technology are 
likely to affect terminal systems presently in use. 


THE TERMINAL OF THE FUTURE IS A 
MINICOMPUTER 

Panelist Robert H. Anderson of The Rand Corporation, 
Santa Monica, California asserts that the single most im¬ 


portant attribute of the interactive computer terminal of the 
future is that it contain a general-purpose computer with the 
approximate characteristics of a present-day minicomputer. 
Representative of the set of services to be provided by the 
terminal of the future is a compatible package of user ser¬ 
vices developed at Rand on a PDP 11/70 minicomputer under 
the UNIX operating system. 1 The major systems within this 
software package (in addition to the standard UNIX ser¬ 
vices) are: a CRT-based two-dimensional text editor 
(NED); 2 an electronic message system (MS); 3 a reminder 
facility (REMIND) capable of sending a message or initiating 
a program at an arbitrary date/time; the RITA system 4-6 for 
creating user agents defined by sets of condition-action 
rules; an extension of the operating system called “virtual 
terminal UNIX” capable of providing the user windows 
onto multiple simultaneous processes; and an interface to 
an electronic data network (in this case, ARPANET). The 
described systems provide a coherent, symbiotic set of ser¬ 
vices that should co-exist. They should be provided on com¬ 
puting power local to the user for several reasons: their 
availability is not subject to external schedules and control; 
access to these services does not require lengthy login and 
authentication protocols to time-shared systems or data net¬ 
works; data privacy can be maintained without reliance on 
central computer system security or transmission lines; pro¬ 
grams within the user’s terminal can act as “agents” capable 
of interacting automatically with external systems on behalf 
of the user, thus providing tailored interfaces and freeing 
the user from mundane, routine interactions. He concludes 
that the most valuable service a user terminal can provide 
is general-purpose computing, both for stand-alone use and 
as an aid in accessing remote information systems. 

THE FUTURE OF SPEECH INPUT TERMINALS 

Another panelist, M. B. Herscher of Threshold Technol¬ 
ogy Inc., Delran, New Jersey, points out that voice data 
entry, long recognized as an ultimate step towards simpli¬ 
fying communications between a human and a machine, is 
now a reality. Over 200 voice terminals currently are in 
operation in a variety of industrial and commercial appli¬ 
cations in eight countries around the world. New speech 
input terminals using microcomputers have been introduced 
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which can replace and/or complement intelligent CRT/key¬ 
board stations by enabling the user to enter data by voice. 
In many applications where hands and/or eyes are occupied 
in normal work environments, voice input terminals permit 
source data to be captured in an extremely efficient manner. 
Since these terminals normally provide input verification 
and editing capabilities, virtually error-free data can be cap¬ 
tured. Speech input terminals also are being used for pro¬ 
gramming NC machines, aids to the handicapped, various 
material handling applications, and in a variety of training 
applications. 

Currently, almost all of these speech input terminals rec¬ 
ognize isolated words or phrases from a limited vocabulary 
subset and require that the user preprogram (train) the sys¬ 
tem for his or her pronunciation of the vocabulary. These 
recognition systems usually require bandwidths of up to 
seven KHz although some of these terminals can operate 
over phone lines. A complementary application of current 
speech input terminals is to recognize (verify identity) of 
individual talkers from their speech characteristics. The first 
secure access system for a computer room using speaker 
authentication has been in operation for several years. 

The potential applications of voice data entry are deter¬ 
mined by the economic advantages of voice input as com¬ 
pared to other alternative data entry devices and/or tech¬ 
niques. As the cost of voice data entry terminals decreases, 
more justifiable applications will arise, particularly when the 
true costs of data capture are considered. In some cases, 
voice input offers advantages which far outweigh the direct 
labor savings and permit certain operations to be achieved 
which could not easily be accomplished using alternative 
data input techniques. 

Almost any system using intelligent CRT/keyboard ter¬ 
minals for data input could use speech input terminals. Vir¬ 
tually any factor or office system which is now directed by 
instructions from either a keyboard or a computer could be 
controlled by verbal commands. With current speech tech¬ 
nology, the human voice simply replaces the mechanical 
keyboard or computer control, often with distinct opera¬ 
tional and economic advantages. 

Mr. Herscher expects the progress in the future for speech 
input terminals to take several directions. Larger size vo¬ 
cabularies will become practical for isolated word recogni¬ 
tion systems. Some systems will be developed which will 
not require the user to pre-train the system vocabulary, 
although higher recognition accuracies always will be 
achievable for the adaptive system. Reliable operation over 
switched phone lines will open up new applications of 
speech terminals. Continuous speech recognition terminals 
will be available within the next one to five years—initially, 
connected digit recognition, followed by larger vocabularies. 
Combined recognition/speaker verification terminals will be 
available for accessing computer data bases and for elec¬ 
tronic funds transfer, eventually over the phone lines. 

He concludes with the observation that additional prog¬ 
ress will be needed to achieve practical continuous speech 
recognition. Difficulty will mount in a geometric progression 
as more sophisticated speech systems are attempted. Once 
the cost barriers are overcome, the benefits will mount 


quickly, so the results promise to be worth the effort and 
investment. Over a ten year period, the first terminals for 
recognizing relatively natural speech input will begin to ap¬ 
pear. 


KEYBOARD DATA ENTRY ISN’T DEAD YET 

The third panelist at this session, Carolyn Dunning of 
PCC Business Systems, Santa Ana, California addresses the 
impacts of technology advances on current data entry ter¬ 
minals and their use. How long does it take for an incumbent 
industry or profession to yield to the onslaught of new ad¬ 
vances? She observes that keyboard data entry is supposed 
to have been ‘dying’ for over 20 years, ever since the intro¬ 
duction of optical character recognition (OCR). Each time 
a new device is developed the cry “keyboard data entry will 
become obsolete in five years” is heard again. Yet through 
all of this, there are more data entry keyboards of some type 
in use today, and more are projected for the future. 

Special purpose terminals such as point of sale (POS) and 
new means for data entry such as voice recognition, may 
indeed have an impact on traditional data entry. But it is 
unlikely that it will be very great. Special purpose terminals 
are too specialized, and so is voice data entry (which is also 
too slow for high volume processing). 

Distributed processing and distributed data entry may 
have a big impact on the traditional centralized data entry 
concept. However, most distributed data entry still uses a 
keyboard of some type. While it may look more like a 
typewriter, it will still use a keyboard. The emphasis may 
not be on quantity, or production, but on timeliness and 
accuracy which will be increased by having the person most 
familiar with the data entering it. 

According to forecasts prepared by International Data 
Corporation (IDC), machine oriented systems such as OCR 
will experience approximately a 5 percent growth between 
1976 and 1981. Keyboard type data entry will see an increase 
of nearly 100 percent, and this takes into account an esti¬ 
mated 16 percent decrease of centralized data entry systems, 
such as key punch and key-to-disk. 

The predicted growth will come from increased installa¬ 
tions of distributed systems, both single station and clus¬ 
tered as well as increased use of conversational and editing 
terminals. All these employ keyboard devices for data entry. 

Another indication that keyboard data entry is still very 
much alive, even in a centralized environment, is the short¬ 
age of data entry operators. The newspapers are full of ads 
for these people. In addition, many companies are offering 
a reward for anyone who sends a qualified applicant for data 
entry. Other companies are taking more trainees or training 
people themselves. When operators can be found, they cost 
more. In the decade from 1966 to 1976, the average salaries 
for key entry operators in all grades increased by more than 
50 percent. 

While certain types of data can be entered in a ‘dumb’ 
sense, exactly as seen, the majority will still take some 
device capable of thinking and making decisions. The best 
device for this is still the human brain. Working in conjunc- 
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tion with the eyes, and fingers, it is the most accurate, rapid 
and practical method of entering data available, not just 
now, but for many years to come. 
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The future of special purpose terminals 


by D. G. KOVAR 

TRW Communications Systems and Services 
Torrance, California 


INTRODUCTION 

In his book, Future Shock, Alvin Toffler dramatized the fact 
that we live in a rapidly changing world, in which most of 
us must constantly struggle to merely stay current. With this 
thought in mind, attempts at predicting the future must al¬ 
ways be approached with some trepidation and perhaps a 
dash of humility. Nevertheless, it is the intent of this article 
to postulate the future of Special Purpose Terminals over 
the period of the next 10 years (1978-1988). This will be 
done by assessing the present state of this field, examining 
current trends, and extrapolating these findings to produce 
a range of future expectations. Although the year 1984 falls 
within the period of study, it is the opinion of the author 
that “Big Brother” will not be watching you on a Special 
Purpose Terminal during this time. 

For the purposes of this paper, we shall define the term 
Special Purpose Terminal to mean computer input/output 
devices which are used by people and which have physical 
and/or functional characteristics that are specialized for a 
particular application or environment. For example, using 
this definition, we would conclude that a laser scanning, 
supermarket point of sale unit integrated with the check 
stand 1 is a Special Purpose Terminal, but on the contrary, 
a Teletype is not. There are, of course, many examples in 
which the line between special and general purpose is less 
clear than the above. Consequently, the definition given is 
useful as a guideline, but not as a rigid rule. Using this 
definition guideline, we find the following application areas 
to be currently active in the field of Special Purpose Ter¬ 
minals: 2 

Point-of-Sale 

Retail Department and Specialty Stores 
Drug and Discount Stores 
Supermarket 
Service Station 

Banking 

Teller Assist 
Automated Tellers 
EFT 


IndustriallMedical 

Factory Data Collection 
Hospital 

Security Access Control 
Rapid Transit 
Parking Lot 

Travel and Entertainment 
Airline Ticketing and Reservation 
Rental Car 

Entertainment Ticketing 

Automated Traveler’s Check Dispensers 

Miscellaneous 

Library 

Law Enforcement 

As time goes on, this already impressive list will be strength¬ 
ened and some potentially very large new categories will be 
added. 3 These are as follows: 

Office Automation 
Electronic Mail 
Conferencing 

Distributed Work Environment 
Consumer Applications 

The requirement for existence of any of these terminal 
devices (both current and future) is primarily a question of 
economics. Advances in technology have made it possible 
to produce increasingly sophisticated terminals at improving 
costs and reliability. Yet, in order for equipment to be com¬ 
mercially produced, there must be an adequate demand for 
the functions provided. Specifically, there must be a large 
enough separation between the economic value (real or per¬ 
ceived) and the cost of production in order to motivate 
investment by both the supplier and the buyer. This can be 
visualized as shown in Figure 1. 

The relationships of Figure 1 must hold. Otherwise, the 
product will not be produced even if it is desired, or it will 
not be sold even if it is produced, or it will be produced and 
sold for a time period until either buyer or supplier with- 
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^ Buyer's Motivation 


^ Supplier's Motivation 


Figure 1—Product viability 


drawal occurs. Consequently, it is insufficient to examine 
the future by studying only the technology. One must ex¬ 
amine both sides of the equation and attempt to project 
product instances where satisfactory value/cost relation¬ 
ships will be likely. This is the methodology used in this 
paper. 


TECHNOLOGY 

Dramatic advances in technology have made the modem 
Special Purpose Terminal a very powerful subsystem. Many 
of the terminals of today already use one or more micropro¬ 
cessors 4 to control the logical operation and have several 
10’s of K Bytes of programmable storage. Having this com¬ 
pute power in the terminal allows for such functions as 
operator prompting, input data validation, transaction-ori¬ 
ented arithmetic, interactive communication with a higher- 
level computer, and complete stand-alone operation if ne¬ 
cessitated by communication failure. This compute power 
has been made available primarily by advances in the state 
of the LSI art. 5 It is expected that such advances will con¬ 
tinue in this area leading to the realm of VLSI. 6 This will in 
turn make it practical to house substantially more compute 
power in the terminal than is possible today. However, 
before commenting on the significance of this, let us first 
take a slightly broader look at the technological needs of the 
Special Purpose Terminal. 

The Special Purpose Terminal is primarily a person-com¬ 
puter input/output device. The basic methods used to per¬ 
form the input/output functions are shown in Figure 2. As 
can be seen, the variety of input/output means is very large 
and each of these means involves its own peculiar technol¬ 
ogy. It should be further noted that only a few of these 
means directly benefit from advances in LSI technology. 
Consequently, so far during the 70’s, we have experienced 
considerable disparity between the rate of change of tech¬ 
nology that is used for the internal logic vis-a-vis that which 
is used for the input/output functions. 7,8 

The result has been a Special Purpose Terminal price 


versus time curve that has been almost flat and, in some 
cases, has even risen slightly. At the same time however, 
logical capability (or performance, if you will) has risen in 
direct proportion to LSI progress, thereby producing signif¬ 
icant gains in the performance/price ratio. So, one could 
characterize the experience to date as being essentially flat 
on price with rising performance rather than fixed in per¬ 
formance at declining prices. In order to be economically 
viable, such a situation must be accompanied by a rising 
demand for performance coupled with a tolerance for con¬ 
stant price. So far, this has been the case. 

Whether or not we can extrapolate from this constant 
price, rising performance, profile is a key question in pro¬ 
jecting the future. The rising performance part seems a safe 
bet, so let’s examine that first. Depending upon the geo¬ 
metric regularity, current production level MOS LSI could 
be described as having reached the 10-20 thousand elements/ 
chip level of density. Advances in mask making technology 
such as electron beam methods promise to raise that density 
to the one million-plus level during the period of study. 9 In 
anticipation of these improvements, the term VLSI is com¬ 
ing into popular use. This suggests that we can expect 32- 
bit computers, one million-bit memories, and combinations 
of these on individual chips. 10 Perhaps this will introduce 
the cliche of “the mainframe on a chip.” 

How will we spend that compute power? What good is 
that much horsepower in an individual terminal? The current 
economic picture for the Special Purpose Terminal would 
suggest that the new levels of compute power be spent on 
attacking the input/output problem—both for the operator 
and and the programmer. Increased compute power can 
improve the economics of some of the input/output devices 
by absorbing more and more of the special electronic pro¬ 
cessing that is externally contained in current equipment— 
i.e., laser scanner, OCR wand, etc. Increased compute 
power can be used to make new input/output methods pos¬ 
sible—i.e., fingerprint, signature, voice, etc. The program¬ 
mability of the Special Purpose Terminal is a very important 
feature in providing the adaptability and flexibility needed 
to meet changing requirements. The value of this feature 
should be expected to increase as time goes on. Conse¬ 
quently, “wasting” some of these new levels of compute 
power and memory space on higher level, more person-like 
(less machine-like) programming languages and tools can be 
anticipated. 11 So, we should look forward to Special Purpose 
Terminals with some increased function and with substan¬ 
tially improved person-machine interfaces. 

In addition to the improvements expected in the basic 
logic function which are spurred on by advances in LSI, we 
can also see some considerable advances in the local storage 
function. Local storage is used primarily to collect data 
generated by the terminal for subsequent transmission and 
to store programs used by the terminal. Normally such stor¬ 
age must be non-volatile, and this requirement is currently 
satisfied by using small disc or tape units. The magnetic 
bubble memory 12,13 is an attractive alternative technology. 
Just now being commercially introduced, the bubble mem¬ 
ory is expected to surpass these electromechanical devices 
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in performance/price ratio’s by 1980. High expectations for 
continuing bubble memory density improvement beyond 
1980 are realistic because such improvements are dependent 
upon the same improvements in mask making technology as 
is the case for LSI. Therefore, bubble memory is expected 
to be used as direct replacement for current local storage 
systems. In addition, the constantly improving bubble mem¬ 
ory economics (ala LSI —> VLSI) will make it possible to 
deploy non-volatile local storage in terminal environments 
for which such storage is not currently cost-effective. 

As previously mentioned, while dramatic advances in 
some portions of the Special Purpose Terminal technology 
can be expected, there are some items that resist. Among 
the technologies implied by Figure 2 displays, printers, 
packaging and power supplies stand out as the most un¬ 
yielding. While many new technologies have been intro¬ 
duced for small terminal displays such as liquid crystal, 
plasma, and LED, they are still not economic for displays 
of several hundred characters. 14 In fact, compared to CRT, 
the disparity can approach an order of magnitude. 

Because of this difference, and because of the pressure 
(for improved person-machine interfaces) which will empha¬ 
size larger displays (including graphics), the CRT will con¬ 
tinue as a major display technology. This projection is made 
by assuming that a “breakthrough” based upon some cur¬ 
rently unknown technology does not take place. Although 
several printer technologies are being pursued, 15 i.e., impact 
dot-matrix, thermal, electro-static, ink-jet, 16 etc., they all 
have moving mechanical parts and suffer from relatively 
high initial cost and continuing reliability problems. The 
printer has sometimes been described as the “Achilles’ 
Heel” of the Special Purpose Terminal. However, some 
cost improvement will be realized by minimizing or com¬ 
pletely avoiding the printing requirement through the use of 
solid state technology as an alternative method for infor¬ 
mation processing. For example, local storage by bubble 
memory can eliminate the journal and information distribu¬ 
tion by electronic means can offset the need for printing on 
multi-part forms. The printer will not go away, however, as 
there will be a continuing need for some hard copy for an 
indefinite future period. In the packaging and power supply 
technologies we can expect some improvement via superior 
materials, components, and design architecture but we 
should not count on these advances producing any more 
than a hedge against inflation. As a result of the above 
observations, the constant price phenomenon is expected to 
continue for some time except for those gains that can be 
realized by substituting solid state for printed information 
handling. 

The picture being painted is one of constantly improving 
performance and functionality under a fairly constant or, at 
best, modestly declining price umbrella. We can expect 
those 1988 Special Purpose Terminals to be programmable 
in very high level languages, to provide solid state, non¬ 
volatile local storage, to offer highly economic input media 
readers, and to introduce some elementary versions of al¬ 
ternate human input such as voice, fingerprint, etc. But 
where will this technology be used? What actual products 
can be expected? Let us turn to that now. 


DEMAND 

Today we find Special Purpose Terminals well entrenched 
in the major retail department stores and financial institu¬ 
tions. 17,18 The demand for these terminals has come from 
the values derived from the improved automated assist func¬ 
tions for clerks and tellers and from the improved velocity 
and accuracy of information associated with consumer 
transactions in the institutions. Such values can be ex¬ 
pressed in terms of improved productivity, asset manage¬ 
ment, and customer service. In a society confronted with a 
growing rate of change, these values can be expected to 
continuously increase. Therefore, we project a continuing 
demand for Special Purpose Terminals by the retail and 
financial institutions. Here, we will see heavy emphasis on 
improving the programmability to keep up with the rate of 
change, and in moving more and more to automated input 
means as the economics improve. Some customer finger¬ 
print optical processing for identification may come into 
use but demand for widespread use of voice input/output is 
doubtful in the active lobby-like atmosphere of these envi¬ 
ronments. 

The question of demand for Special Purpose Terminals 
for use in consumer transaction electronic funds transfer 
(EFT) is fascinating in its complexity. 19 Here is a case in 
which the necessary technology already exists. But who is 
the terminal buyer? Who benefits? Who runs the multi-bank, 
multi-retailer interconnect network? Can terminals be legally 
shared? What about banking legislation in general? What 
about consumer protection legislation? Satisfactory answers 
to these questions will be needed before significant invest¬ 
ments are made in EFT. 20 

In spite of these problems, there are some strong moti¬ 
vators for EFT. The banking institutions face the same rising 
costs of real estate, and energy as are pervasive in society 
in general. In addition, the growing volume of paper in the 
system is a constant threat to processing saturation. On a 
more positive note, EFT should be able to deliver a more 
expeditious and broader range of financial services to con¬ 
sumers. 

So, how will this tug-of-war be decided? It is projected 
here that consumer transaction EFT will come about but it 
will be a slow process. Many experiments by individual 
banking institutions have been conducted and more will 
follow. 22 The anticipated progress toward true distributed 
financial institution EFT is projected to still be struggling 
for definition by 1980 and just beginning to take shape by 
1985. Again, note that this is not a terminal technology 
problem. Consequently, the EFT Special Purpose Terminal 
product will still be quite young in 1988 23 and by then the 
EFT function will be fully integrated into the retail POS 
terminal and is not expected to exist as a separate entity. 

Many of the forces (high rate of change, rising personnel 
costs, energy, etc.) that contribute to the demand found in 
retail and financial institutions are present in other trans¬ 
action environments as well. Therefore, a similar continuing 
demand is expected in factory data collection, hospital pa¬ 
tient administration, hotel and airline reservation, and other 
similar situations. 
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Perhaps the most exciting trends regarding demand are 
found as we explore the migration of special purpose ter¬ 
minals from the world of transaction control, to the envi¬ 
ronment of office automation, 24 to the surroundings of our 
own homes. Here, the rising costs of energy are continuing 
to raise the value associated with moving electrons instead 
of moving physical media or people. 25 Interestingly enough, 
consumer infatuation with calculators, home computers, 
and TV games could thrust the Special Purpose Terminal 
into the home more quickly than will be the case for the 
office. In fact, this phenomenon will tend to blur the dis¬ 
tinction between what is office and what is home. One can 
foresee a complete, home-based, Special Purpose Terminal 
system with one or more stations for entertainment, for 
education, for work, and for shopping and banking trans¬ 
actions. Furthermore, such a system could control the home 
environment and optimize energy consumption in its spare 
time. 

It is reasonable to expect that such equipment will enter 
the home first on the entertainment motive and then incor¬ 
porate the other functions incrementally as the necessary 
supporting systems (i.e., EFT, education, etc.) come into 
being. It is projected that by 1985, the bulk of the terminal 
equipment will be available but the community-level sup¬ 
porting systems will be very primitive. By 1988 the distrib¬ 
uted, home-based terminal environment will have positioned 
itself for growth on into the 1990’s. 


CONCLUSION 

We have painted a picture of rising demand and of contin¬ 
uing technological advance which would seem to imply dra¬ 
matic growth. However, this is offset in those instances 
where substantial utility-like supporting systems are needed 
which must cross institutional lines. It doesn’t do any good 
to possess a very sophisticated person-machine interface 
terminal if there is no distribution system. Therefore, growth 
is expected; it will be exciting, but except for those pockets 
of explosive growth that can move ahead on an isolated 
system basis, the progress will be slower than we might 
otherwise expect. Nevertheless, the Special Purpose Ter¬ 
minal world of 1988 will offer a substantial marketplace for 
the producer, powerful information systems for the business 
user, and some real, quality-of-life benefits for the con¬ 
sumer. One of the great entertainment values of the 1980’s 


is going to consist of following the progress of the Special 
Purpose Terminal. Be sure not to miss it. 
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Office automation 


Daily, new uses appear for computers in the general office environment. While 
the area is still dominated by word processing, other applications are becoming 
widespread. Special systems for administrative tasks, records management, doc¬ 
ument distribution, and communication integrate new activities into the standard 
word processing and data processing activities. 

With these new applications, however, come endless problems—discovering 
and refining the applications, choosing equipment from the constant new offer¬ 
ings, devising appropriate accounting methods, restructuring organizations, and 
determining the impact on the corporation—all are complex issues. 

The first session will review the history and current state of office automation. 
The evolutionary changes of the field from the technical and marketing point of 
view will be reviewed. The history of the industry will be developed and will be 
compared to the data processing industry. Finally, the surprising diversity of the 
office automation industry will be outlined with a review of the range and appli¬ 
cations and the present areas of concentration. 

The architecture of systems for office automation is diverse and rapidly chang¬ 
ing. The second session will present three views of these architectural changes; 
firmware to software, stand-alone to distributed, and maxi to micro. While we 
observed these changes initially in hardware, they greatly impact the applications. 
The long range user effects will be the focus of the session. 

The third session will address organizational trends in office automation, and 
will cover methodologies for designing automated office support. They include: 
integration into computer-based, multi-function terminals for total automated 
support, cost/benefit analysis for managerial work, increased user participation in 
the design and revision of automated office systems, and increased attention to 
organizational and human factors in system design and implementation. 

Even with these methodologies, however, the implementation of automated 
office systems has been difficult. Cost-benefit data is confusing, and overall 
productivity gains are difficult to determine. This fourth session will present a 
different, controversial position. 

Office automation systems imposed on existing organizational information 
flows will increase inter-group conflict, confuse assignments, cloud responsibility 
and provide no overall improvement in performance. If office automation only 
provides faster switching of messages and easier access to files it will increase 
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the problems inherent in current information flows and be rejected by the users. 
What is needed is formal reconstituting of information flows by the major activ¬ 
ities of data collection, data base analysis, information packaging and knowledge 
acceptance. The delineation of basic processes will make possible scheduling and 
planning models, resource allocation, and assignment of overall objectives and 
responsibility. Unless these improvements are made in information processes 
office automation will not realize its full potential. 

Finally, some users have begun to reconstitute their information flows and 
address the issues raised in the previous session. This final, fifth, session focuses 
on several applications of computer-based message and document distribution 
systems. Emphasis is on implementation approaches and system value within the 
organizational context, and on the introduction of such systems to prospective 
users. Case presentations will be made in which three organizations are repre¬ 
sented—one multi-national non-profit, one corporate, and one federal government 
agency. Each has actually implemented a system and will discuss its usage and 
related impacts. 




Area Director: 

Robert Balzer 

Information Science Institute 
Marina del Rey, California 


Artificial intelligence 


The bulk of current Artificial Intelligence research is focused on methods for 
achieving goals within partially understood environments. In such environments 
the bodies of knowledge which exist are quite varied; some are systematic and 
well validated, others are judgemental. Characteristically, there are no known 
optimal ways of applying the knowledge in the environment to achieve the desired 
goals. The challenge of A1 is to develop methods for handling problems in such 
environments and to aid the development of a better understanding of these 
environments, which may in turn lead to more powerful ways of achieving goals 
within it. 

The development of Artificial Intelligence has been marked by two rather 
distinct periods. In the first, the basic structure of the field emerged, centered on 
the notion of general purpose mechanisms and general-purpose problem solvers, 
theorem provers, and semantic nets. Once the limitations of such general purpose 
mechanisms were generally recognized, the second period of development began. 
It was, and is, marked by the attempt to incorporate and effectively utilize special 
purpose knowledge appropriate for some task. Issues such as acquiring such 
special purpose knowledge from experts, representing it in a useful form, and 
effectively using it to control a problem solving activity became paramount. 

These NCC sessions will focus on this latter period through two invited papers. 
The first, presented by Professor Edward Feigenbaum, chronicles the develop¬ 
ment of the specialized knowledge period through the activities of the Stanford 
Heuristic Programming Project which has long been a leader in the development 
of expert-level performance programs. The second invited paper is the lecture 
presented by Professor Douglas Lenat upon his selection for the 1977 Computers 
and Thought Lecture for outstanding contributions by a young researcher to 
Artificial Intelligence. Lenat’s talk and research describe a system for discovering 
new mathematical concepts and conjectures involving those concepts. This fas¬ 
cinating work indicates the potential feasibility of providing operational models 
of the process of mathematical discovery. 

These two invited papers, which comprise the first A1 session, will surely 
initiate a rash of questions. Therefore, they will immediately be followed by a 
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session entirely devoted to answering questions from the audience on these and 
other A1 issues. Joining Feigenbaum and Lenat on the panel of A1 experts will be 
Professors Raj Reddy and Saul Amarel, and Dr. Peter Hart. The final session 
will present several state-of-the-art applications which have achieved expert-level 
performance. 



The art of artificial intelligence—Themes and case studies of 
knowledge engineering 

by EDWARD A. FEIGENBAUM 

Stanford University 
Stanford, California 


INTRODUCTION—AN EXAMPLE 

This paper will examine emerging themes of knowledge en¬ 
gineering, illustrate them with case studies drawn from the 
work of the Stanford Heuristic Programming Project, and 
discuss general issues of knowledge engineering art and 
practice. 

Let me begin with an example new to our workbench: a 
system called PUFF, the early fruit of a collaboration be¬ 
tween our project and a group at the Pacific Medical Center 
(PMC) in San Francisco.* 

A physician refers a patient to PMC’s pulmonary function 
testing lab for diagnosis of possible pulmonary function dis¬ 
order. For one of the tests, the patient inhales and exhales 
a few times in a tube connected to an instrument/computer 
combination. The instrument acquires data on flow rates 
and volumes, the so-called flow-volume loop of the patient’s 
lungs and airways. The computer measures certain param¬ 
eters of the curve and presents them to the diagnostician 
(physician or PUFF) for interpretation. The diagnosis is 
made along these lines: normal or diseased; restricted lung 
disease or obstructive airways disease or a combination of 
both; the severity; the likely disease type(s) (e.g., emphy¬ 
sema, bronchitis, etc.); and other factors important for di¬ 
agnosis. 

PUFF is given not only the measured data but also certain 
items of information from the patient record, e.g., sex, age, 
number of pack-years of cigarette smoking. The task of the 
PUFF system is to infer a diagnosis and print it out in 
English in the normal medical summary form of the inter¬ 
pretation expected by the referring physician. 

Everything PUFF knows about pulmonary function di¬ 
agnosis is contained in (currently) 55 rules of the IF. . . 
THEN. . . form. No textbook of medicine currently records 
these rules. They constitute the partly-public, partly-private 
knowledge of an expert pulmonary physiologist at PMC, and 
were extracted and polished by project engineers working 
intensively with the expert over a period of time. Here is an 
example of a PUFF rule (the unexplained acronyms refer to 
various data measurements): 


* Dr. J. Osborn, Dr. R. Fallat, John Kunz, Diane McClung. 


RULE 31 
IF: 

1) The severity of obstructive airways 
disease of the patient is greater than or 
equal to mild, and 

2) The degree of diffusion defect of the 
patient is greater than or equal to mild, 
and 

3) The tic (body box) observed/predicted of 
the patient is greater than or equal to 110 
and 

4) The observed-predicted difference in 
rv/tlc of the patient is greater than or 
equal to 10 

THEN: 

1) There is strongly suggestive evidence 
(.9) that the subtype of obstructive airways 
disease is emphysema, and 

2) It is definite (1.0) that “OAD, 

Diffusion Defect, elevated TLC, and elevated 
RV together indicate emphysema.” is one of 
the findings. 

One hundred cases, carefully chosen to span the variety 
of disease states with sufficient exemplary information for 
each, were used to extract the 55 rules. As the knowledge 
emerged, it was represented in rule form, added to the sys¬ 
tem and tested by running additional cases. The expert was 
sometimes surprised, sometimes frustrated, by the occa¬ 
sional gaps and inconsistencies in the knowledge, and the 
incorrect diagnoses that were logical consequences of the 
existing rule set. The interplay between knowledge engineer 
and expert gradually expanded the set of rules to remove 
most of these problems. 

As cumulation of techniques in the art demands and al¬ 
lows, a new tool was not invented when an old one would 
do. The knowledge engineers pulled out of their toolkit a 
version of the MYCIN system (to be discussed later), with 
the rules about infectious diseases removed, and used it as 
the inference engine for the PUFF diagnoses. Thus PUFF, 
like MYCIN, is a relatively simple backward-chaining infer- 


227 



228 


National Computer Conference, 1978 


ence system. It seeks a valid line-of-reasoning based on its 
rules and rooted in the instrument and patient data. With a 
little more work at fitting some existing tools together, 
PUFF will be able to explain this line-of-reasoning, just as 
MYCIN does. 

As it is, PUFF only prints out the final interpretation, of 
which the following is an example: 

PATIENT DATA: 

The degree of dyspnea: MODERATELY-SEVERE 

The severity of coughing: MILD 

Sputum production MODERATELY-SEVERE 

The number of pack-years of smoking: 48 

referral diagnosis: BRONCHITIS 

IVC/IVC-predicted: 80 

RV/RV-predicted: 191 

FVC/FVC-predicted: 87 

TLC(body boxjobserved/predicted: 127 

Predicted FEV1/FVC: 83 

TLC(DLCO)observed/predicted: 83 

FEV1/FVC ratio: 50 

RV/TLC Observed-Predicted: 21 

MMF/MMF-predicted: 19 

the DLCO/DLCO-predicted: 48 

The slope (F50_obs-F25_obs)/FVC_obs: 19 

DEGREE OF OBSTRUCTIVE AIRWAYS DISEASE: 

OAD degree by SLOPE: (MODERATELY-SEVERE 
700) 

OAD degree by MMF: (SEVERE 900) 

OAD degree by FEV1: (MODERATELY-SEVERE 
700) 

FINAL OAD DEGREE: (MODERATELY-SEVERE 
910) (SEVERE 900) 

No conflict. Final degree: 

(MODERATELY-SEVERE 910) 

INTERPRETATION: 

Obstruction is indicated by curvature of 
the flow-volume loop. 

Forced Vital Capacity is normal and peak 
flow rates are reduced, suggesting 
airway obstruction. 

Flow rate from 25-75 of expired volume is 
reduced, indicating severe airway 
obstruction. 

OAD, Diffusion Defect, elevated TLC, and 
elevated RV together indicate emphysema. 

OAD, Diffusion Defect, and elevated RV 
indicate emphysema. 

Change in expired flow rates following 
bronchodilation shows that there is 
reversibility of airway obstruction. 

The presence of a productive cough is an 
indication that the OAD is of the 
bronchitic type. 

Elevated lung volumes indicate 
overinflation. 

Air trapping is indicated by the elevated 


difference between observed and predicted 
RV/TLC ratios. 

Improvement in airway resistance indicates 
some reversibility of airway 
Airway obstruction is consistent with the 
patient’s smoking history. 

The airway obstruction accounts for the 
patient’s dyspnea. 

Although bronchodilators were not 
useful in this one case, prolonged use may 
prove to be beneficial to the patient. 

The reduced diffusion capacity indicates 
airway obstruction of the mixed 
bronchitic and emphysematous types. 

Low diffusing capacity indicates loss of 
alveolar capillary surface. 

Obstructive Airways Disease of mixed types 

150 cases not studied during the knowledge acquisition 
process were used for a test and validation of the rule set. 
PUFF inferred a diagnosis for each. PUFF-produced and 
expert-produced interpretations were coded for statistical 
analysis to discover the degree of agreement. Over various 
types of disease states, and for two conditions of match 
between human and computer diagnoses (“same degree of 
severity” and “within one degree of severity”), agreement 
ranged between approximately 90 percent and 100 percent. 

The PUFF story is just beginning and will be told perhaps 
at a later NCC. The surprising punchline to my synopsis is 
that the current state of the PUFF system as described 
above was achieved in less than 50 hours of interaction with 
the expert and less than 10 man-weeks of effort by the 
knowledge engineers. We have learned much in the past 
decade of the art of engineering knowledge-based intelligent 
agents! 

In the remainder of this essay, I would like to discuss the 
route that one research group, the Stanford Heuristic Pro¬ 
gramming Project, has taken, illustrating progress with case 
studies, and discussing themes of the work. 

ARTIFICIAL INTELLIGENCE & KNOWLEDGE 
ENGINEERING 

The dichotomy that was used to classify the collected 
papers in the volume Computers and Thought still charac¬ 
terizes well the motivations and research efforts of the AI 
community. First, there are some who work toward the 
construction of intelligent artifacts, or seek to uncover prin¬ 
ciples, methods, and techniques useful in such construction. 
Second, there are those who view artificial intelligence as 
(to use Newell’s phrase) “theoretical psychology,” seeking 
explicit and valid information processing models of human 
thought. 

For purposes of this essay, I wish to focus on the moti¬ 
vations of the first group, these days by far the larger of the 
two. I label these motivations “the intelligent agent view¬ 
point” and here is my understanding of that viewpoint: 

“The potential uses of computers by people to accom- 
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plish tasks can be ‘one-dimensionalized’ into a spectrum 
representing the nature of instruction that must be given 
the computer to do its job. Call it the WHAT-to-HOW 
spectrum. At one extreme of the spectrum, the user sup¬ 
plies his intelligence to instruct the machine with precision 
exactly HOW to do his job, step-by-step. Progress in 
Computer Science can be seen as steps away from the 
extreme ‘HOW’ point on the spectrum: the familiar pan¬ 
oply of assembly languages, subroutine libraries, compil¬ 
ers, extensible languages, etc. At the other extreme of 
the spectrum is the user with his real problem (WHAT he 
wishes the computer, as his instrument, to do for him). 
He aspires to communicate WHAT he wants done in a 
language that is comfortable to him (perhaps English); via 
communication modes that are convenient for him (in¬ 
cluding perhaps, speech or pictures); with some general¬ 
ity, some vagueness, imprecision, even error; without 
having to lay out in detail all necessary subgoals for ad¬ 
equate performance—with reasonable assurance that he 
is addressing an intelligent agent that is using knowledge 
of his world to understand his intent, to fill in his vague¬ 
ness, to make specific his abstractions, to correct his 
errors, to discover appropriate subgoals, and ultimately 
to translate WHAT he really wants done into processing 
steps that define HOW it shall be done by a real computer. 
The research activity aimed at creating computer pro¬ 
grams that act as “intelligent agents” near the WHAT 
end of the WHAT-To-HOW spectrum can be viewed as 
the long-range goal of AI research.” (Feigenbaum, 1974) 

Our young science is still more art than science. Art: “the 
principles or methods governing any craft or branch of learn¬ 
ing.” Art: “skilled workmanship, execution, or agency.” 
These the dictionary teaches us. Knuth tells us that the 
endeavor of computer programming is an art, in just these 
ways. The art of constructing intelligent agents is both part 
of and an extension of the programming art. It is the art of 
building complex computer programs that represent and rea¬ 
son with knowledge of the world. Our art therefore lives in 
symbiosis with the other worldly arts, whose practitioners— 
experts of their art—hold the knowledge we need to con¬ 
struct intelligent agents. In most “crafts or branches of 
learning” what we call “expertise” is the essence of the art. 
And for the domains of knowledge that we touch with our 
art, it is the “rules of expertise” or the rules of “good 
judgment” of the expert practitioners of that domain that 
we seek to transfer to our programs. 

Lessons of the past 

Two insights from previous work are pertinent to this 
essay. 

The first concerns the quest for generality and power of 
the inference engine used in the performance of intelligent 
acts (what Minsky and Papert [see Goldstein and Papert, 
1977] have labeled “the power strategy”). We must hypoth¬ 
esize from our experience to date that the problem solving 
power exhibited in an intelligent agent’s performance is pri¬ 


marily a consequence of the specialist’s knowledge em¬ 
ployed by the agent, and only very secondarily related to 
the generality and power of the inference method employed. 
Our agents must be knowledge-rich, even if they are meth- 
ods-poor. In 1970, reporting the first major summary-of- 
results of the DENDRAL program (to be discussed later), 
we addressed this issue as follows: 

“. . . general problem-solvers are too weak to be used 
as the basis for building high-performance systems. The 
behavior of the best general problem-solvers we know, 
human problem-solvers, is observed to be weak and shal¬ 
low, except in the areas in which the human problem- 
solver is a specialist. And it is observed that the transfer 
of expertise between specialty areas is slight. A chess 
master is unlikely to be an expert algebraist or an expert 
mass spectrum analyst, etc. In this view, the expert is the 
specialist, with a specialist’s knowledge of his area and a 
specialist’s methods and heuristics.” (Feigenbaum, Buch¬ 
anan and Lederberg, 1971, p. 187) 

Subsequent evidence from our laboratory and all others 
has only confirmed this belief. 

AI researchers have dramatically shifted their view on 
generality and power in the past decade. In 1967, the can¬ 
onical question about the DENDRAL program was: “It 
sounds like good chemistry, but what does it have to do 
with AI?” In 1977, Goldstein and Papert write of a paradigm 
shift in AI: 

“Today there has been a shift in paradigm. The fun¬ 
damental problem of understanding intelligence is not the 
identification of a few powerful techniques, but rather the 
question of how to represent large amounts of knowledge 
in a fashion that permits their effective use and interac¬ 
tion.” (Goldstein and Papert, 1977). 

The second insight from past work concerns the nature of 
the knowledge that an expert brings to the performance of 
a task. Experience has shown us that this knowledge is 
largely heuristic knowledge, experiential, uncertain—mostly 
“good guesses” and “good practice,” in lieu of facts and 
rigor. Experience has also taught us that much of this knowl¬ 
edge is private to the expert, not because he is unwilling to 
share publicly how he performs, but because he is unable. 
He knows more than he is aware of knowing. [Why else is 
the Ph.D. or the Internship a guild-like apprenticeship to a 
presumed “master of the craft?” What the masters really 
know is not written in the textbooks of the masters.] But 
we have learned also that this private knowledge can be 
uncovered by the careful, painstaking analysis of a second 
party, or sometimes by the expert himself, operating in the 
context of a large number of highly specific performance 
problems. Finally, we have learned that expertise is multi¬ 
faceted, that the expert brings to bear many and varied 
sources of knowledge in performance. The approach to cap¬ 
turing his expertise must proceed on many fronts simulta¬ 
neously. 




230 


National Computer Conference, 1978 


The knowledge engineer 

The knowledge engineer is that second party just dis¬ 
cussed. She works intensively with an expert to acquire 
domain-specific knowledge and organize it for use by a pro¬ 
gram. Simultaneously she is matching the tools of the AI 
workbench to the task at hand—program organizations, 
methods of symbolic inference, techniques for the structur¬ 
ing of symbolic information, and the like. If the tool fits, or 
nearly fits, she uses it. If not, necessity mothers AI inven¬ 
tion, and a new tool gets created. She builds the early ver¬ 
sions of the intelligent agent, guided always by her intent 
that the program eventually achieve expert levels of per¬ 
formance in the task. She refines or reconceptualizes the 
system as the increasing amount of acquired knowledge 
causes the AI tool to “break” or slow down intolerably. 
She also refines the human interface to the intelligent agent 
with several aims: to make the system appear “comforta¬ 
ble” to the human user in his linguistic transactions with it; 
to make the system’s inference processes understandable to 
the user; and to make the assistance controllable by the user 
when, in the context of a real problem, he has an insight 
that previously was not elicited and therefore not incorpo¬ 
rated. 

In the next section, I wish to explore (in summary form) 
some case studies of the knowledge engineer’s art. 

CASES FROM THE KNOWLEDGE ENGINEER’S 
WORKSHOP 

I will draw material for this section from the work of my 
group at Stanford. Much exciting work in knowledge engi¬ 
neering is going on elsewhere. Since my intent is not to 
survey literature but to illustrate themes, at the risk of ap¬ 
pearing parochial I have used as case studies the work I 
know best. 

My collaborators (Professors Lederberg and Buchanan) 
and I began a series of projects, initially the development of 
the DENDRAL program, in 1965. We had dual motives: 
first, to study scientific problem solving and discovery, par¬ 
ticularly the processes scientists do use or should use in 
inferring hypotheses and theories from empirical evidence; 
and second, to conduct this study in such a way that our 
experimental programs would one day be of use to working 
scientists, providing intelligent assistance on important and 
difficult problems. By 1970, we and our co-workers had 
gained enough experience that we felt comfortable in laying 
out a program of research encompassing work on theory 
formation, knowledge utilization, knowledge acquisition, 
explanation, and knowledge engineering techniques. Al¬ 
though there were some surprises along the way, the general 
lines of the research are proceeding as envisioned. 


THEMES 

As a road map to these case studies, it is useful to keep 
in mind certain major themes: 


Generation-and-test: Omnipresent in our experiments is the 
“classical” generation-and-test framework that has been 
the hallmark of AI programs for two decades. This is not 
a consequence of a doctrinaire attitude on our part about 
heuristic search, but rather of the usefulness and suffi¬ 
ciency of the concept. 

Situation^Action Rules: We have chosen to represent the 
knowledge of experts in this form. Making no doctrinaire 
claims for the universal applicability of this representa¬ 
tion, we nonetheless point to the demonstrated utility of 
the rule-based representation. From this representation 
flow rather directly many of the characteristics of our 
programs: for example, ease of modification of the knowl¬ 
edge, ease of explanation. The essence of our approach 
is that a rule must capture a “chunk” of domain knowl¬ 
edge that is meaningful, in and of itself, to the domain 
specialist. Thus our rules bear only a historical relation¬ 
ship to the production rules used by Newell and Simon 
(1972) which we view as “machine-language program¬ 
ming” of a recognize=>act machine. 

The Domain-Specific Knowledge: It plays a critical role in 
organizing and constraining search. The theme is that in 
the knowledge is the power. The interesting action arises 
from the knowledge base, not the inference engine. We 
use knowledge in rule form (discussed above), in the form 
of inferentially-rich models based on theory, and in the 
form of tableaus of symbolic data and relationships (i.e., 
frame-like structures). System processes are made to con¬ 
form to natural and convenient representations of the do- 
main-specific knowledge. 

Flexibility to modify the knowledge base: If the so-called 
“grain size” of the knowledge representation is chosen 
properly (i.e., small enough to be comprehensible but 
large enough to be meaningful to the domain specialist), 
then the rule-based approach allows great flexibility for 
adding, removing, or changing knowledge in the system. 

Line-of-reasoning: A central organizing principle in the de¬ 
sign of knowledge-based intelligent agents is the mainte¬ 
nance of a line-of-reasoning that is comprehensible to the 
domain specialist. This principle is, of course, not a logical 
necessity, but seems to us to be an engineering principle 
of major importance. 

Multiple Sources of Knowledge: The formation and main¬ 
tenance (support) of the line-of-reasoning usually require 
the integration of many disparate sources of knowledge. 
The representational and inferential problems in achieving 
a smooth and effective integration are formidable engi¬ 
neering problems. 

Explanation: The ability to explain the line-of-reasoning in 
a language convenient to the user is necessary for appli¬ 
cation and for system development (e.g., for debugging 
and for extending the knowledge base). Once again, this 
is an engineering principle, but very important. What con- 
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stitutes “an explanation” is not a simple concept, and 

considerable thought needs to be given, in each case, to 

the structuring of explanations. 

CASE STUDIES 

In this section I will try to illustrate these themes with 
various case studies. 

DENDRAL: Inferring chemical structures 

Historical note 

Begun in 1965, this collaborative project with the Stanford 
Mass Spectrometry Laboratory has become one of the long¬ 
est-lived continuous efforts in the history of AI (a fact that 
in no small way has contributed to its success). The basic 
framework of generation-and-test and rule-based represen¬ 
tation has proved rugged and extendable. For us the DEN¬ 
DRAL system has been a fountain of ideas, many of which 
have found their way, highly metamorphosed, into our other 
projects. For example, our long-standing commitment to 
rule-based representations arose out of our (successful) at¬ 
tempt to head off the imminent ossification of DENDRAL 
caused by the rapid accumulation of new knowledge in the 
system around 1967. 

Task 

To enumerate plausible structures (atom-bond graphs) for 
organic molecules, given two kinds of information: analytic 
instrument data from a mass spectrometer and a nuclear 
magnetic resonance spectrometer; and user-supplied con¬ 
straints on the answers, derived from any other source of 
knowledge (instrumental or contextual) available to the user. 

Representations 

Chemical structures are represented as node-link graphs 
of atoms (nodes) and bonds (links). Constraints on search 
are represented as subgraphs (atomic configurations) to be 
denied or preferred. The empirical theory of mass spectrom¬ 
etry is represented by a set of rules of the general form: 

Situation: Particular atomic 
configuration 
(subgraph) 
i 

I 

| Probability, P, 
j of occurring 

I 

I 

I 

V 

Action: Fragmentation of the 

particular configuration 
(Breaking links) 


Rules of this form are natural and expressive to mass 
spectrometrists. 


Sketch of method 

DENDRAL’s inference procedure is a heuristic search 
that takes place in three stages, without feedback: plan- 
generate-test. 

“Generate” (a program called CONGEN) is a generation 
process for plausible structures. Its foundation is a combi¬ 
natorial algorithm (with mathematically proven properties of 
completeness and non-redundant generation) that can pro¬ 
duce all the topologically legal candidate structures. Con¬ 
straints supplied by the user or by the “Plan” process prune 
and steer the generation to produce the plausible set (i.e., 
those satisfying the constraints) and not the enormous legal 
set. 

“Test” refines the evaluation of plausibility, discarding 
less worthy candidates and rank-ordering the remainder for 
examination by the user. “Test” first produces a “pre¬ 
dicted” set of instrument data for each plausible candidate, 
using the rules described. It then evaluates the worth of 
each candidate by comparing its predicted data with the 
actual input data. The evaluation is based on heuristic cri¬ 
teria of goodness-of-fit. Thus, “test” selects the “best” 
explanations of the data. 

“Plan” produces direct (i.e., not chained) inference about 
likely substructure in the molecule from patterns in the data 
that are indicative of the presence of the substructure. (Pat¬ 
terns in the data trigger the left-hand-sides of substructure 
rules). Though composed of many atoms whose intercon¬ 
nections are given, the substructure can be manipulated as 
atom-like by “generate.” Aggregating many units entering 
into a combinatorial process into fewer higher-level units 
reduces the size of the combinatorial search space. “Plan” 
sets up the search space so as to be relevant to the input 
data. “Generate is the inference tactician; “Plan” is the 
inference strategist. There is a separate “Plan” package for 
each type of instrument data, but each package passes sub¬ 
structures (subgraphs) to “Generate.” Thus, there is a uni¬ 
form interface between “Plan” and “Generate.” User-sup¬ 
plied constraints enter this interface, directly or from user- 
assist packages, in the form of substructures. 


Sources of knowledge 

The various sources of knowledge used by the DEN¬ 
DRAL system are: 

Valences (legal connections of atoms); stable and un¬ 
stable configurations of atoms; rules for mass spectrom¬ 
etry fragmentations; rules for NMR shifts; experts’ rules 
for planning and evaluation; user-supplied constraints 
(contextual). 
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Results 

DENDRAL’s structure elucidation abilities are, paradox¬ 
ically, both very general and very narrow. In general, DEN- 
DRAL handles all molecules, cyclic and tree-like. In pure 
structure elucidation under constraints (without instrument 
data), CONGEN is unrivaled by human performance. In 
structure elucidation with instrument data, DENDRAL’s 
performance rivals expert human performance only for a 
small number of molecular families for which the program 
has been given specialist’s knowledge, namely the families 
of interest to our chemist collaborators. I will spare this 
computer science audience the list of names of these fami¬ 
lies. Within these areas of knowledge-intensive specializa¬ 
tion, DENDRAL’s performance is usually not only much 
faster but also more accurate than expert human perform¬ 
ance. 

The statement just made summarizes thousands of runs 
of DENDRAL on problems of interest to our experts, their 
colleagues, and their students. The results obtained, along 
with the knowledge that had to be given to DENDRAL to 
obtain them, are published in major journals of chemistry. 
To date, 25 papers have been published there, under a series 
title “Applications of Artificial Intelligence for Chemical 
Inference: (specific subject)” (see for example, the Bu¬ 
chanan, Smith, et al., 1976, reference). 

The DENDRAL system is in everyday use by Stanford 
chemists, their collaborators at other universities and col¬ 
laborating or otherwise interested chemists in industry. 
Users outside Stanford access the system over commercial 
computer/communications network. The problems they are 
solving are often difficult and novel. The British government 
is currently supporting work at Edinburgh aimed at trans¬ 
ferring DENDRAL to industrial user communities in the 
UK. 

Discussion 

Representation and extensibility. The representation cho¬ 
sen for the molecules, constraints, and rules of instrument 
data interpretation is sufficiently close to that used by chem¬ 
ists in thinking about structure elucidation that the knowl¬ 
edge base has been extended smoothly and easily, mostly 
by chemists themselves in recent years. Only one major 
reprogramming effort took place in the last 9 years—when 
a new generator was created to deal with cyclic structures. 

Representation and the Integration of multiple sources of 
knowledge. The generally difficult problem of integrating 
various sources of knowledge has been made easy in DEN¬ 
DRAL by careful engineering of the representations of ob¬ 
jects, constraints, and rules. We insisted on a common lan¬ 
guage of compatibility of the representations with each other 
and with the inference processes: the language of molecular 
structure expressed as graphs. This leads to a straightfor¬ 
ward procedure for adding a new source of knowledge, say, 
for example, the knowledge associated with a new type of 
instrument data. The procedure is this: write rules that de¬ 
scribe the effect of the physical processes of the instrument 
on molecules using the situation=>action form with molec¬ 


ular graphs on both sides; any special inference process 
using these rules must pass its results to the generator only 
(!) in the common graph language. 

It is today widely believed in AI that the use of many 
diverse sources of knowledge in problem solving and data 
interpretation has a strong effect on quality of performance. 
How strong is, of course, domain-dependent, but the impact 
of bringing just one additional source of knowledge to bear 
on a problem can be startling. In one difficult (but not un¬ 
usually difficult) mass spectrum analysis problem,* the pro¬ 
gram using its mass spectrometry knowledge alone would 
have generated an impossibly large set of plausible candi¬ 
dates (over 1.25 million!). Our engineering response to this 
was to add another source of data and knowledge, proton 
NMR. The addition on a simple interpretive theory of this 
NMR data, from which the program could infer a few ad¬ 
ditional constraints, reduced the set of plausible candidates 
to one, the right structure! This was not an isolated result 
but showed up dozens of times in subsequent analyses. 

DENDRAL and data. DENDRAL’s robust models (top¬ 
ological, chemical, instrumental) permit a strategy of find¬ 
ing solutions by generating hypothetical “correct answers” 
and choosing among these with critical tests. This strategy 
is opposite to that of piecing together the implications of 
each data point to form a hypothesis. We call DENDRAL’s 
strategy largely model-driven, and the other data-driven. 
The consequence of having enough knowledge to do model- 
driven analysis is a large reduction in the amount of data 
that must be examined since data is being used mostly for 
verification of possible answers. In a typical DENDRAL 
mass spectrum analysis, usually no more than about 25 data 
points out of a typical total of 250 points are processed. This 
important point about data reduction and focus-of-attention 
has been discussed before by Gregory (1968) and by the 
vision and speech research groups, but is not widely under¬ 
stood. 

Conclusion. DENDRAL was an early herald of AI’s shift 
to the knowledge-based paradigm. It demonstrated the point 
of the primacy of domain-specific knowledge in achieving 
expert levels of performance. Its development brought to 
the surface important problems of knowledge representa¬ 
tion, acquisition, and use. It showed that, by and large, the 
AI tools of the first decade were sufficient to cope with the 
demands of a complex scientific problem-solving task, or 
were readily extended to handle unforeseen difficulties. It 
demonstrated that AI’s conceptual and programming tools 
were capable of producing programs of applications interest, 
albeit in narrow specialties. Such a demonstration of com¬ 
petence and sufficiency was important for the credibility of 
the AI field at a critical juncture in its history. 

META-DENDRAL: inferring rules of mass spectrometry 

Historical note 

The META-DENDRAL program is a case study in auto¬ 
matic acquisition of domain knowledge. It arose out of our 


* The analysis of an acyclic amine with formula C20H45N. 
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DENDRAL work for two reasons: first, a decision that with 
DENDRAL we had a sufficiently firm foundation on which 
to pursue our long-standing interest in processes of scientific 
theory formation; second, by a recognition that the acqui¬ 
sition of domain knowledge was the bottleneck problem in 
the building of applications-oriented intelligent agents. 


Task 

META-DENDRAL’s job is to infer rules of fragmentation 
of molecules in a mass spectrometer for possible later use 
by the DENDRAL performance program. The inference is 
to be made from actual spectra recorded from known mo¬ 
lecular structures. The output of the system is the set of 
fragmentation rules discovered, summary of the evidence 
supporting each rule, and a summary of contra-indicating 
evidence. User-supplied constraints can also be input to 
force the form of rules along desired lines. 


Representations 

The rules are, of course, of the same form as used by 
DENDRAL that was described earlier. 


Sketch of method 

META-DENDRAL, like DENDRAL, uses the genera- 
tion-and-test framework. The process is organized in three 
stages: Reinterpret the data and summarize evidence 
(INTSUM); generate plausible candidates for rules (RU- 
LEGEN); test and refine the set of plausible rules (RULE- 
MOD). 

INTSUM: gives every data point in every spectrum an 
interpretation as a possible (highly specific) fragmentation. 
It then summarizes statistically the “weight of evidence” 
for fragmentations and for atomic configurations that cause 
these fragmentations. Thus, the job of INTSUM is to trans¬ 
late data to DENDRAL subgraphs and bond-breaks, and to 
summarize the evidence accordingly. 

RULEGEN: conducts a heuristic search of the space of 
all rules that are legal under the DENDRAL rule syntax and 
the user-supplied constraints. It searches for plausible rules, 
i.e., those for which positive evidence exists. A search path 
is pruned when there is no evidence for rules of the class 
just generated. The search tree begins with the (single) most 
general rule (loosely put, “anything” fragments from “any¬ 
thing”) and proceeds level-by-level toward more detailed 
specifications of the “anything.” The heuristic stopping cri¬ 
terion measures whether a rule being generated has become 
too specific, in particular whether it is applicable to too few 
molecules of the input set. Similarly there is a criterion for 
deciding whether an emerging rule is too general. Thus, the 
output of RULEGEN is a set of candidate rules for which 
there is positive evidence. 

RULEMOD: tests the candidate rule set using more com¬ 


plex criteria, including the presence of negative evidence. 
It removes redundancies in the candidate rule set; merges 
rules that are supported by the same evidence; tries further 
specialization of candidates to remove negative evidence; 
and tries further generalization that preserves positive evi¬ 
dence. 


Results 

META-DENDRAL produces rule sets that rival in quality 
those produced by our collaborating experts. In some tests, 
META-DENDRAL re-created rule sets that we had previ¬ 
ously acquired from our experts during the DENDRAL proj¬ 
ect. In a more stringent test involving members of a family 
of complex ringed molecules for which the mass spectral 
theory had not been completely worked out by chemists, 
META-DENDRAL discovered rule sets for each subfamily. 
The rules were judged by experts to be excellent and a paper 
describing them was recently published in a major chemical 
journal (Buchanan, Smith, et al, 1976). 

In a test of the generality of the approach, a version of 
the META-DENDRAL program is currently being applied 
to the discovery of rules for the analysis of nuclear magnetic 
resonance data. 


MYC1N and TEIRESIAS: Medical diagnosis 

Historical note 

MYCIN originated in the Ph.D. thesis of E. Shortliffe 
(now Shortliffe, M.D. as well), in collaboration with the 
Infectious Disease group at the Stanford Medical School 
(Shortliffe, 1976). TEIRESIAS, the Ph.D. thesis work of R. 
Davis, arose from issues and problems indicated by the 
MYCIN project but generalized by Davis beyond the bounds 
of medical diagnosis applications (Davis, 1976). Other 
MYCIN-related theses are in progress. 

Tasks 

The MYCIN performance task is diagnosis of blood in¬ 
fections and meningitis infections and the recommendation 
of drug treatment. MYCIN conducts a consultation (in Eng¬ 
lish) with a physician-user about a patient case, constructing 
lines-of-reasoning leading to the diagnosis and treatment 
plan. 

The TEIRESIAS knowledge acquisition task can be de¬ 
scribed as follows: 

In the context of a particular consultation, confront the 
expert with a diagnosis with which he does not agree. Lead 
him systematically back through the line-of-reasoning that 
produced the diagnosis to the point at which he indicates 
the analysis went awry. Interact with the expert to modify 
offending rules or to acquire new rules. Rerun the consul¬ 
tation to test the solution and gain the expert’s concurrence. 
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Representations: 

MYCIN’s rules are of the form: 

IF (conjunctive clauses) THEN (implication) 

Here is an example of a MYCIN rule for blood infections. 

RULE 85 
IF: 

1) The site of the culture is blood, and 

2) The gram stain of the organism is 
gramneg, and 

3) The morphology of the organism is 
rod, and 

4) The patient is a compromised host 
THEN: 

There is suggestive evidence (.6) that 
the identity of the organism is 
pseudomonas-aeruginosa 

TEIRESIAS allows the representation of MYCIN-like 
rules governing the use of other rules, i.e., rule-based strat¬ 
egies. An example follows. 

METARULE 2 
IF: 

1) the patient is a compromised host, and 

2) there are rules which mention in their 
premise pseudomonas 

3) there are rules which mention in their 
premise klebsiellas 

THEN: 

There is suggestive evidence (.4) that the 
former should be done before the latter. 


Sketch of method 

MYCIN employs a generation-and-test procedure of a 
familiar sort. The generation of steps in the line-of-reasoning 
is accomplished by backward chaining of the rules. An IF- 
side clause is either immediately true or false (as determined 
by patient or test data entered by the physician in the con¬ 
sultation); or is to be decided by subgoaling. Thus, “test” 
is interleaved with “generation” and serves to prune out 
incorrect lines-of-reasoning. 

Each rule supplied by an expert has associated with it a 
“degree of certainty” representing the expert’s confidence 
in the validity of the rule (a number from 1 to 10). MYCIN 
uses a particular ad-hoc but simple model of inexact reason¬ 
ing to cumulate the degrees of certainty of the rules used in 
an inference chain (Shortliffe and Buchanan, 1975). 

It follows that there may be a number of “somewhat true” 
lines-of-reasoning—some indicating one diagnosis, some in¬ 


dicating another. All (above a threshold) are used by the 
system as sources of knowledge indicating plausible lines- 
of-reasoning. 

TEIRESIAS’ rule acquisition process is based on a record 
of MYCIN’s search. Rule acquisition is guided by a set of 
rule models that dictate the form and indicate the likely 
content of new rules. Rule models are not given in advance, 
but are inferred from the knowledge base of existing rules. 


Language of interaction 

The language used looks like it might be English but is 
actually the dialect “Doctor-ese” used by members of the 
tribe of healing arts practitioners. Reasonably simple lan¬ 
guage processing methods suffice. When ambiguities of in¬ 
terpretation are encountered, they are fed back to the user 
for decisions about meaning. 


Example diagnosis and treatment plan 

The following is an example of a diagnosis reported by 
MYCIN to one of its users following a consultation. An 
example consultation will not be shown, but one can be 
found in Shortliffe’s book (Shortliffe, 1976). 

My therapy recommendations will be designed 
to treat for organisms that are either very 
likely or, although less likely, would have 
a significant effect on therapy selection 
if they were present. It is important to 
cover for the following probable 
infection(s) and associated organism(s): 

INFECTION-1 is CYSTITIS 

(Item 1) PSEUDOMONAS-CEPACIA [ORGANISM-5] 
(Item 2) CITROBACTER-DIVERSUS [ORGANISM-4] 
(Item 3) E.COLI [ORGANISM-3] 

INFECTION-2 is UPPER-RESPIRATORY- 
INFECTION 

(Item 4) LISTERIA [ORGANISM-2] 

INFECTION-3 is BACTEREMIA 

(Item 5) ENTEROCOCCUS [ORGANISM-1] 

[REC-1] My preferred therapy recommendation 
is as follows: 

In order to cover for Items (1 2 3 4 5): 

Give the following in combination: 

1) KANAMYCIN 

Dose: 750 mg (7.5 mg/kg) ql2h IM (or IV) 
for 28 days 

Comments: Modify dose in renal failure 

2) PENICILLIN 

Dose: 2,500,000 units (25000 units/kg) 
q4h IV for 28 days 
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Results 

No clinical evaluation of MYCIN is currently being done. 
In a preliminary evaluation, done outside of the clinical 
setting, a panel of expert judges was asked to rate MYCIN’s 
performance along a number of dimensions (e.g., infecting 
organism identification, organism significance). In 90 per¬ 
cent of the cases submitted to the judges, a majority of the 
judges said that the program’s decisions were the-same-as 
or as-good-as the decisions they would have made. 

Explanation 

MYCIN can expose its line of reasoning by offering ex¬ 
planations of various kinds. “Why” and “How” questions 
can be asked by the user during the consultation. “Why” 
is the user’s way of inquiring, “Why do you want to know 
that?” “How” is his way of asking, “How did you conclude 
that. . . ?” In a post-consultation, he can similarly inquire 
about how the diagnostic conclusions were reached, and 
inquire about the reasons why certain other conclusions 
were ruled out. Examples of the MYCIN-TEIRESIAS ex¬ 
planation capabilities follow: 

MYCIN: Is the patient’s illness with 
ORGANISM-1 a hospital-acquired infection? 

USER: WHY 

TEIRESIAS: (i.e., WHY is it important to 
determine whether or not the infection with 
ORGANISM-1 was acquired while the patient 
was hospitalized?) 

[1.0] This will aid in determining the 
category of ORGANISM-1. 

It has already been established that 

[1.1] the gram stain of ORGANISM-1 is 
gramneg, and 

[1.2] the morphology of ORGANISM-1 is rod, 
and 

[1.3] the aerobicity of ORGANISM-1 is 
facultative 

Therefore, if 

[1.4] the infection with ORGANISM-1 was not 
acquired while the patient was hospitalized 

then 

there is strongly suggestive evidence (.8) 
that the category of ORGANISM-1 is 
enterobacteriaceae 

Also: there is weakly suggestive evidence 
(.1) that the identity of 


ORGANISM-1 is pseudomonas-aeruginosa 
[RULE 37] 

USER: HOW 1.3 

TEIRESIAS: [i.e. HOW was it established 
that the aerobicity of ORGANISM-1 is 
facultative?] 

The following were used: 

[3.1] RULE 27 indicated there is strongly 
suggestive evidence (.8) that the 
aerobicity of ORGANISM-1 is facultative 

[3.2] RULE 27 indicated there is weakly 
suggestive evidence (.2) that the 
aerobicity of ORGANISM-1 is anaerobic 

Since this gave a cumulative CF of (.8) for 
facultative, and (.2) for anaerobic, it has 
been established that the aerobicity of 
ORGANISM-1 is facultative. 

USER: WHY DIDN’T YOU CONSIDER 
STREPTOCOCCUS AS A POSSIBILITY FOR 
ORGANISM-1? 

TEIRESIAS: The following rule could have 
been used to determine that the identity of 
ORGANISM-1 was streptococcus: RULE 33 

But clause 2 (“the morphology of the 
organism is coccus”) was already known 
to be false for ORGANISM-1, so the rule was 
never tried. 


Discussion 

A few short comments aimed at knowledge engineering 
are in order. 

English is the language most natural and comfortable for 
our experts and users, and was therefore the language cho¬ 
sen for interactive consultation, explanation, and external 
representation of the rules (the internal format is INTER¬ 
LISP). This situation is not peculiar to doctors; in most 
areas of application of intelligent agents I believe that Eng¬ 
lish (i.e., natural language) will be the language of choice. 
Programming an English language processor and front-end 
to such systems is not a scary enterprise because: 

(a) the domain is specialized, so that possible interpreta¬ 
tions are constrained. 

(b) specialist-talk is replete with standard jargon and ster¬ 
eotyped ways of expressing knowledge and queries—just 
right for text templates, simple grammars and other simple 
processing schemes. 
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(c) the ambiguity of interpretation resulting from simple 
schemes can be dealt with easily by feeding back interpre¬ 
tations for confirmation. If this is done with a pleasant “I 
didn’t quite understand you. . tone, it is not irritating to 
the user. 

English may be exactly the wrong language for represen¬ 
tation and interaction in some domains. It would be awk¬ 
ward, to say the least, to represent DENDRAL’s chemical 
structures and knowledge of mass spectrometry in English, 
or to interact about these with a user. 

Simple explanation schemes have been a part of the AI 
scene for a number of years and are not hard to implement. 
Really good models of what explanation is as a transaction 
between user and agent, with programs to implement these 
models, will be the subject (1 predict) of much future re¬ 
search in AI. 

Without the explanation capability, I assert, user accept¬ 
ance of MYCIN would have been nil, and there would have 
been a greatly diminished effectiveness and contribution of 
our experts. 

MYCIN was the first of our programs that forced us to 
deal with what we had always understood: that experts’ 
knowledge is uncertain and that our inference engines had 
to be made to reason with this uncertainty. It is less impor¬ 
tant that the inexact reasoning scheme be formal, rigorous, 
and uniform than it is for the scheme to be natural to and 
easily understandable by the experts and users. 

All of these points can be summarized by saying that 
MYCIN and its TEIRESIAS adjunct are experiments in the 
design of a see-through system, whose representations and 
processes are almost transparently clear to the domain spe¬ 
cialist. “Almost” here is equivalent to “with a few minutes 
of introductory description.” The various pieces of 
MYCIN—the backward chaining, the English transactions, 
the explanations, etc.—are each simple in concept and re¬ 
alization. But there are great virtues to simplicity in system 
design; and viewed as a total intelligent agent system, 
MYCIN/TEIRESIAS is one of the best engineered. 

SU/X: signal understanding 

Historical note 

SU/X is a system design that was tested in an application 
whose details are classified. Because of this, the ensuing 
discussion will appear considerably less concrete and tan¬ 
gible than the preceding case studies. This system design 
was done by H. P. Nii and me, and was strongly influenced 
by the CMU Hearsay II system design (Lesser and Erman, 
1977). 


Task 

SU/X’s task is the formation and continual updating, over 
long periods of time, of hypotheses about the identity, lo¬ 
cation, and velocity of objects in a physical space. The 
output desired is a display of the “current best hypotheses” 


with full explanation of the support for each. There are two 
types of input data: the primary signal (to be understood); 
and auxiliary symbolic data (to supply context for the un¬ 
derstanding). The primary signals are spectra, represented 
as descriptions of the spectral lines. The various spectra 
cover the physical space with some spatial overlap. 

Representations 

The rules given by the expert about objects, their behav¬ 
ior, and the interpretation of signal data from them are all 
represented in the situation=>action form. The “situations” 
constitute invoking conditions and the “actions” are pro¬ 
cesses that modify the current hypotheses, post unresolved 
issues, recompute evaluations, etc. The expert’s knowledge 
of how to do analysis in the task is also represented in rule 
form. These strategy rules replace the normal executive 
program. 

The situation-hypothesis is represented as a node-link 
graph, tree-like in that it has distinct “levels,” each repre¬ 
senting a degree of abstraction (or aggregation) that is nat¬ 
ural to the expert in his understanding of the domain. A 
node represents an hypothesis; a link to that node represents 
support for that hypothesis (as in HEARSAY II, “support 
from above” or “support from below”). “Lower” levels 
are concerned with the specifics of the signal data. “Higher” 
levels represent symbolic abstractions. 

Sketch of method 

The situation-hypothesis is formed incrementally. As the 
situation unfolds over time, the triggering of rules modifies 
or discards existing hypotheses, adds new ones, or changes 
support values. The situation-hypothesis is a common work¬ 
space (“blackboard,” in HEARSAY jargon) for all the rules. 

In general, the incremental steps toward a more complete 
and refined situation-hypothesis can be viewed as a se¬ 
quence of local generate-and-test activities. Some of the 
rules are plausible move generators, generating either nodes 
or links. Other rules are evaluators, testing and modifying 
node descriptions. 

In typical operation, new data is submitted for processing 
(say, N time-units of new data). This initiates a flurry of 
rule-triggerings and consequently rule-actions (called 
“events”). Some events are direct consequences of data; 
other events arise in a cascade-like fashion from the trig¬ 
gering of rules. Auxiliary symbolic data also cause events, 
usually affecting the higher levels of the hypothesis. As a 
consequence, support-from-above for the lower level pro¬ 
cesses is made available; and expectations of possible lower 
level events can be formed. Eventually all the relevant rules 
have their say and the system becomes quiescent, thereby 
triggering the input of new data to reenergize the inference 
activity. 

The system uses the simplifying strategy of maintaining 
only one “best” situation-hypothesis at any moment, mod¬ 
ifying it incrementally as required by the changing data. This 
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approach is made feasible by several characteristics of the 
domain. First, there is the strong continuity over time of 
objects and their behaviors (specifically, they do not change 
radically over time, or behave radically differently over 
short periods). Second, a single problem (identity, location 
and velocity of a particular set of objects) persists over 
numerous data gathering periods. (Compare this to speech 
understanding in which each sentence is spoken just once, 
and each presents a new and different problem.) Finally, the 
system’s hypothesis is typically “almost right,” in part be¬ 
cause it gets numerous opportunities to refine the solution 
(i.e., the numerous data gathering periods), and in part be¬ 
cause the availability of many knowledge sources tends to 
over-determine the solution. As a result of all of these, the 
current best hypothesis changes only slowly with time, and 
hence keeping only the current best is a feasible approach. 

Of interest are the time-based events. These rule-like 
expressions, created by certain rules, trigger upon the pas¬ 
sage of specified amounts of time. They implement various 
“wait-and-see” strategies of analysis that are useful in the 
domain. 


Results 

In the test application, using signal data generated by a 
simulation program because real data was not available, the 
program achieved expert levels of performance over a span 
of test problems. Some problems were difficult because 
there was very little primary signal to support inference. 
Others were difficult because too much signal induced a 
plethora of alternatives with much ambiguity. 

A modified SU/X design is currently being used as the 
basis for an application to the interpretation of x-ray crys¬ 
tallographic data, the CRYSALIS program mentioned later. 


Discussion 

The role of the auxiliary symbolic sources of data is of 
critical importance. They supply a symbolic model of the 
existing situation that is used to generate expectations of 
events to be observed in the data stream. This allows flow 
of inferences from higher levels of abstraction to lower. 
Such a process, so familiar to AI researchers, apparently is 
almost unrecognized among signal processing engineers. In 
the application task, the expectation-driven analysis is es¬ 
sential in controlling the combinatorial processing explosion 
at the lower levels, exactly the explosion that forces the 
traditional signal processing engineers to seek out the largest 
possible number-cruncher for their work. 

The design of appropriate explanations for the user takes 
an interesting twist in SU/X. The situation-hypothesis un¬ 
folds piecemeal over time, but the “appropriate” explana¬ 
tion for the user is one that focuses on individual objects 
over time. Thus the appropriate explanation must be syn¬ 
thesized from a history of all the events that led up to the 
current hypothesis. Contrast this with the MYCIN-TEI- 


RESIAS reporting of rule invocations in the construction of 
a reasoning chain. 

Since its knowledge base and its auxiliary symbolic data 
give it a model-of-the-situation that strongly constrains in¬ 
terpretation of the primary data stream, SU/X is relatively 
unperturbed by errorful or missing data. These data condi¬ 
tions merely cause fluctuations in the credibility of individ¬ 
ual hypotheses and/or the creation of the “wait-and-see” 
events. SU/X can be (but has not yet been) used to control 
sensors. Since its rules specify what types and values of 
evidence are necessary to establish support, and since it is 
constantly processing a complete hypothesis structure, it 
can request “critical readings” from the sensors. In general, 
this allows an efficient use of limited sensor bandwidth and 
data acquisition processing capability. 


Other case studies 

Space does not allow more than just a brief sketch of 
other interesting projects that have been completed or are 
in progress. 

AM: mathematical discovery 

AM is a knowledge-based system that conjectures inter¬ 
esting concepts in elementary mathematics. It is a discoverer 
of interesting theorems to prove, not a theorem proving 
program. It was conceived and executed by D. Lenat for his 
Ph.D. thesis, and is reported by him in these proceedings. 

AM’s knowledge is basically of two types: rules that sug¬ 
gest possibly interesting new concepts from previously con¬ 
jectured concepts; and rules that evaluate the mathematical 
“interestingness” of a conjecture. These rules attempt to 
capture the expertise of the professional mathematician at 
the task of mathematical discovery. Though Lenat is not a 
professional mathematician, he was able successfully to 
serve as his own expert in the building of this program. 

AM conducts a heuristic search through the space of con¬ 
cepts creatable from its rules. Its basic framework is gen- 
eration-and-test. The generation is plausible move genera¬ 
tion, as indicated by the rules for formation of new concepts. 
The test is the evaluation of “interestingness.” Of particular 
note is the method of test-by-example that lends the flavor 
of scientific hypothesis testing to the enterprise of mathe¬ 
matical discovery. 

Initialized with concepts of elementary set theory, it con¬ 
jectured concepts in elementary number theory, such as 
“add,” “multiply” (by four distinct paths!), “primes,” the 
unique factorization theorem, and a concept similar to 
primes but previously not much studied called “maximally 
divisible numbers.” 

MOLGEN: planning experiments in molecular genetics 

MOLGEN, a collaboration with the Stanford Genetics 
Department, is work in progress. MOLGEN’s task is to 
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provide intelligent advice to a molecular geneticist on the 
planning of experiments involving the manipulation of DNA. 
The geneticist has various kinds of laboratory techniques 
available for changing DNA material (cuts, joins, insertions, 
deletions, and so on); techniques for determining the bio¬ 
logical consequences of the changes; various instruments 
for measuring effects; various chemical methods for induc¬ 
ing, facilitating, or inhibiting changes; and many other tools. 

Some MOLGEN programs under development will offer 
planning assistance in organizing and sequencing such tools 
to accomplish an experimental goal. Other MOLGEN pro¬ 
grams will check user-provided experiment plans for feasi¬ 
bility; and its knowledge base will be a repository for the 
rapidly expanding knowledge of this specialty, available by 
interrogation. 

In MOLGEN the problem of integration of many diverse 
sources of knowledge is central since the essence of the 
experiment planning process is the successful merging of 
biological, genetic, chemical, topological, and instrument 
knowledge. In MOLGEN the problem of representing pro¬ 
cesses is also brought into focus since the expert’s knowl¬ 
edge of experimental strategies—proto-plans—must also be 
represented and put to use. 

One MOLGEN program (Stefik, 1978) solves a type of 
analysis problem that is often difficult for laboratory scien¬ 
tists to solve. DNA structures can be fragmented by chem¬ 
icals called restriction enzymes. These enzymes cut DNA 
at specific recognition sites. The fragmentation may be com¬ 
plete or partial. One or more enzymes may be used. The 
fragmented segments of the DNA are collected and sorted 
out by segment length using a technique called gel electro¬ 
phoresis. The analytical problem is similar to that faced by 
DENDRAL: given an observed fragmentation pattern, hy¬ 
pothesize the best structural explanation of the data. More 
precisely the problem is to map the enzyme recognition sites 
of a DNA structure from complete or partial “digests”. 

The program uses the model-driven approach that is sim¬ 
ilar to DENDRAL’s and is discussed earlier. The method is 
generate-and-test. A generator is initiated that is capable of 
generating all the site-segment maps in an exhaustive, irre- 
dundant fashion. Various pruning rules are used to remove 
whole classes of conceivable candidates in light of the data. 
Some of the pruning rules are empirical and judgmental. 
Others are formal and mathematically based. 

The program solves simpler problems of this type of anal¬ 
ysis better than laboratory scientists. The harder problems, 
however, yield only to the broader biological knowledge 
known by the scientists and not yet available to the pro¬ 
gram’s reasoning processes. In a recent test case, a problem 
whose solution space contained approximately 150,000,000 
site-fragment “maps” was solved in 27 seconds of PDP-10 
time using the INTERLISP programming system. 

Interestingly, the computer scientist’s formal understand¬ 
ing of the nature of the problem, his formal representation 
of the knowledge used for pruning out inappropriate candi¬ 
dates, and the computational power available to him enabled 
him to suggest a few new experiment designs to his geneticist 
collaborators that were not previously in their repertoire. 


CRYSALIS: inferring protein structure from electron 
density maps 


CRYSALIS, too, is work in progress. Its task is to hypoth¬ 
esize the structure of a protein from a map of electron 
density that is derived from x-ray crystallographic data. The 
map is three-dimensional, and the contour information is 
crude and highly ambiguous. Interpretation is guided and 
supported by auxiliary information, of which the amino acid 
sequence of the protein’s backbone is the most important. 
Density map interpretation is a protein chemist’s art. As 
always, capturing this art in heuristic rules and putting it to 
use with an inference engine is the project’s goal. 

The inference engine for CRYSALIS is a modification of 
the SU/X system design described above. The hypothesis 
formation process must deal with many levels of possibly 
useful aggregation and abstraction. For example, the map 
itself can be viewed as consisting of “peaks,” or “peaks 
and valleys,” or “skeleton.” The protein model has 
“atoms,” “amide planes,” “amino acid sidechains,” and 
even massive substructures such as “helices.” Protein mol¬ 
ecules are so complex that a systematic generation-and-test 
strategy like DENDRAL’s is not feasible. Incremental piec¬ 
ing together of the hypothesis using region-growing methods 
is necessary. 

The CRYSALIS design (alias SU/P) is described in a 
recent paper by Nii and Feigenbaum (1977). 

SUMMARY OF CASE STUDIES 

Some of the themes presented earlier need no recapitu¬ 
lation, but I wish to revisit three here: generation-and-test; 
situation=>action rules; and explanations. 

Generation and test 

Aircraft come in a wide variety of sizes, shapes, and 
functional designs and they are applied in very many ways. 
But almost all that fly do so because of the unifying physical 
principle of lift by airflow; the others are described by ex¬ 
ception. If there is such a unifying principle for intelligent 
programs and human intelligence it is generation-and-test. 
No wonder that this has been so thoroughly studied in AI 
research! 

In the case studies, generation is manifested in a variety 
of forms and processing schemes. There are legal move 
generators defined formally by a generating algorithm 
(DENDRAL’s graph generating algorithm); or by a logical 
rule of inference (MYCIN’s backward chaining). When legal 
move generation is not possible or not efficient, there are 
plausible move generators (as in SU/X and AM). Sometimes 
generation is interleaved with testing (as in MYCIN, SU/X, 
and AM). In one case, all generation precedes testing (DEN¬ 
DRAL). One case (META-DENDRAL) is mixed, with some 
testing taking place during generation, some after. 
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Test also shows great variety. There are simple tests 
(MYCIN: “Is the organism aerobic?”; SU/X: “Has a spec¬ 
tral line appeared at position P?”) Some tests are complex 
heuristic evaluations (AM: “Is the new concept ‘interest¬ 
ing’?”; MOLGEN: “Will the reaction actually take place?”) 
Sometimes a complex test can involve feedback to modify 
the object being tested (as in META-DENDRAL). 

The evidence from our case studies supports the assertion 
by Newell and Simon that generation-and-test is a law of 
our science (Newell and Simon, 1976). 

Situation faction rules 

Situation^Action rules are used to represent experts’ 
knowledge in all of the case studies. Always the situation 
part indicates the specific conditions under which the rule 
is relevant. The action part can be simple (MYCIN: con¬ 
clude presence of particular organism; DENDRAL: con¬ 
clude break of particular bond). Or it can be quite complex 
(MOLGEN: an experiential procedure). The overriding con¬ 
sideration in making design choices is that the rule form 
chosen be able to represent clearly and directly what the 
expert wishes to express about the domain. As illustrated, 
this may necessitate a wide variation in rule syntax and 
semantics. 

From a study of all the projects, a regularity emerges. A 
salient feature of the Situation^Action rule technique for 
representing experts’ knowledge is the modularity of the 
knowledge base, with the concomitant flexibility to add or 
change the knowledge easily as the experts’ understanding 
of the domain changes. Here too one must be pragmatic, 
not doctrinaire. A technique such as this cannot represent 
modularity of knowledge if that modularity does not exist in 
the domain. The virtue of this technique is that it serves as 
a framework for discovering what modularity exists in the 
domain. Discovery may feed back to cause reformulation of 
the knowledge toward greater modularity. 

Finally, our case studies have shown that strategy knowl¬ 
edge can be captured in rule form. In TEIRESIAS, the 
metarules capture knowledge of how to deploy domain 
knowledge; in SU/X, the strategy rules represent the ex¬ 
perts’ knowledge of “how to analyze” in the domain. 

Explanation 

Most of the programs, and all of the more recent ones, 
make available an explanation capability for the user, be he 
end-user or system developer. Our focus on end-users in 
applications domains has forced attention to human engi¬ 
neering issues, in particular making the need for the expla¬ 
nation capability imperative. 

The Intelligent Agent viewpoint seems to us to demand 
that the agent be able to explain its activity; else the question 
arises of who is in control of the agent’s activity. The issue 
is not academic or philosophical. It is an engineering issue 
that has arisen in medical and military applications of intel¬ 


ligent agents, and will govern future acceptance of AI work 
in applications areas. And on the philosophical level one 
might even argue that there is a moral imperative to provide 
accurate explanations to end-users whose intuitions about 
our systems are almost nil. 

Finally, the explanation capability is needed as part of the 
concerted attack on the knowledge acquisition problem. Ex¬ 
planation of the reasoning process is central to the interac¬ 
tive transfer of expertise to the knowledge base, and it is 
our most powerful tool for the debugging of the knowledge 
base. 
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INTRODUCTION 

Much of the behavior which we regard as “intelligent” in¬ 
volves some sort of discovery process.** This is obvious for 
some of the most creative and intellectually difficult human 
activities (identifying an unknown chemical compound, 
composing a new sonnet, deriving a new cosmological 
model, ligating plasmids, conjecturing a new theorem, prob¬ 
ing the internal structure of a proton, solving the N. Y. Times 
crossword puzzle, . . .). By the end of this talk we’ll see 
that it’s no less true for our everyday activities (understand¬ 
ing spoken English, making a joke, driving from CMU to 
MIT, cutting cheese, playing board games, finding our way 
about Boston, solving the Pittsburgh Press crossword puz¬ 
zle, . . .). 

Model-building in science 

We in the field of Artificial Intelligence (AI) want to un¬ 
derstand how it’s possible to do such things, to understand 
the mechanisms of intelligence. I’ll propose a theory of how 
humans are able to carry out such tasks, and then embody 
that theory in several concrete, testable models. 

Scientific models are useful because they (i) unify large 
masses of empirical data (e.g., the periodic table); and they 
(ii) predict new effects, which subsequently can be tested 
for by conducting experiments (e.g., Young’s two-slit inter¬ 
ference test for the wave theory of light). 

In very “hard” sciences, objective data are available 
about isolated and relatively simple phenomena. This ena¬ 
bles the construction of small yet quite rigorous predictive 
models, using the language of mathematics (e.g., Maxwell’s 
equations). But in the so-called “soft” fields, the phenom¬ 
ena cannot be measured precisely (the role of the family, 
the “life after life” reports, etc.), or are not so reproducible 
(the evolution of life, the origins of WWI, etc.), or (as in the 
case of human intelligence) cannot be easily isolated for 


* This work was supported in part by the National Science Foundation 
(MCS77-04440), and in part by the Defense Advanced Research Projects 
Agency (F44620-73-C-0074). 

** Much of the rest of “intelligence” involves algorithmic solving of well- 

structured problems. That topic will not be emphasized in this paper. We 
take the position that “algorithms known and used by experts” is just a 
proper subset of “knowledge experts use to reduce search.” 


study (e.g., the genetic component of IQ). The resulting 
models are usually only descriptive, and may often be pre¬ 
sented in everyday prose (e.g., psychological theories of 
personality).*** 

The physicist’s model (his set of equations) is better than 
mere prose because it is able to draw on the power of 
established mathematics! to make quantitative predictions. 

But planets and atoms behave in much more regular a 
fashion than do people. Can we build a hard, predictive sort 
of model for the phenomena of discovery and creativity? Or 
are we limited to sterile prose discussions about the mys¬ 
teries of incubation and illumination (e.g., as in References 
3 and 4)? Can we draw only metaphorical pictures (e.g., as 
in the ‘hooked atoms’ image that Einstein conjured up when 
pressed to introspect on how he attained his insights; 5 or as 
in the “snake swallowing its own tail” dream that inspired 
Kekule to consider that the structure of the benzine mole¬ 
cule might be a ring)l The answer, until just twenty years 
ago, was unfortunately “We can’t do any better.” When the 
answer changed, a whole new science, Artificial Intelli¬ 
gence, was bom. 

Choosing an approach in science 

Each science is differentiated from the others not merely 
by the set of phenomena it claims as its object of study, but 
also by the approach it takes (the science’s view of those 
phenomena; its paradigm , 6 if you will). 

So even though we’ve decided to study the phenomena of 
human intelligence (creativity, problem solving, concept for¬ 
mation, etc.), we must still choose a view of Man. 

If we view Man as an actor whose internal thought pro¬ 
cesses can’t be investigated, then we are called “behavioral 
psychologists,” and we study human behavior. If we view 
Man as a brain, as a piece of hardware built out of neurons, 
then we’re called “biologists” and we study neurophysio- 


*** Attempts to formalize “soft” phenomena do go on continually, but the 
interpretation of the resultant rigorous mathematics is often a topic of heated 
debate (a current example is Catastrophe Theory. 1 

t The power to solve analysis problems in closed form (e.g., to solve a 
differential equation simply by repeatedly manipulating it according to known 
transformations), or the power to make approximations when necessary, or 
the power to somehow “run” the model (to “grind out” solutions to his 
equations). 
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SOME HEURISTICS ARE ALMOST ALWAYS RELEVANT 

(a) > If X is often true, try to find out exactly where it does and doesn’t hold. 

(b) >If you must do some new, complicated task, try to arrange things 

so that the tools, subtasks, etc. are very familiar. 

>Look at the extreme cases of the known relationships. 

>Ignore minor details until a basic plan is formed. 


SOME HEURISTICS ARE RELEVANT WHEN DOING MATH OR SCIENCE 

>Even if a theory predicts something strongly, 

if a very easy test is available, do it (you may be surprised). 

>The more important a step is (in a proof, an experiment, etc.), 
the more stringent verification to which it should be subjected. 

>Design experiments so that a negative result will still be interesting. 

>When preparing an article’s bibliography, cite all possible referees. 


SOME HEURISTICS ARE RELEVANT 
WHEN DOING BIOLOGY 

(a) >It’s interesting to study whether a 

mechanism from one species also 
operates in another species. 

(b) >When choosing which type of 

experimental animal to use, favor a 
species about which much is already 
known. 


SOME HEURISTICS ARE RELEVANT 
WHEN DOING MOLECULAR GENETICS 

(a) >It’s interesting to study whether gene 

control signals from one species are 
operative in another species. 

(b) >E. Coli is a favorite because much is 

known about its genetics; many of its 
plasmids have been characterized and 
are available. 

>If you want to introduce DNA between 
strains of bacteria, then typical vectors 
to use are plasmids and lysogenic 
viruses. 

>To check whether a gene transferred to a 
new host has been modified, reintroduce 
it into its donor. 


SOME HEURISTICS ARE RELEVANT 

WHEN DOING MATHEMATICS 

>If f: A x A .v ... x A - B, and A <= B, 

then see if in fact f: A x A .. . > A -> A; 

Else try to find a subset S of A 
for which f : SxSx.. .xS - - S. 

>Ii' a set S has only a few elements, 
then it’s no longer of interest; 

But it is worth trying to understand 
the reason why S is so small. 


Figure 1—Hierarchy of heuristic rules 


logical responses (e.g., by implanting electrodes). If we view 
Man as a machine, as an automaton, then we’re called “cy¬ 
berneticists,” and we investigate mathematical properties of 
feedback networks of simple components. If we view Man 
as a collection of atomic particles, then we’re called fool¬ 
ish,** for this is too fine a “granularity” with which to 
investigate intelligence. 

Another view arose about twenty years ago, from three 


** Unlike the Brownian motion of atoms in a perfect gas, the fundamental 
information processes of intelligence are not random. 


independent sources: (0 engineering (Donald Broadbent: 
Man can be viewed as an information processor with a 
limited memory system), (//) psycholinguistics (Noam 
Chomsky: Grammar characterizes human competence; 
hence, Man can be viewed as a system governed by abstract 
rules), and (in') computer science and cognitive psychology 
(Allen Newell and Herbert Simon: Man can be viewed as a 
processor of symbols). If we adopt that view of Man, then 
we’re working in the field of “Artificial Intelligence.” 

No one view of Man is “right” or “wrong;” each is 
adopted because from it we can build a model, which in turn 
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has some practical consequences and uses. When I’m ill, I 
want to go to a doctor who practices medicine based on 
Man as a biological organism, not Man as an economic 
agent. At the present limited state of our understanding, any 
one view is bound to be simplistic and incomplete.* It is the 
whole man who lives and reacts, even though we can only 
view him, first this way, then that. 

The foundation for artificial intelligence 

Suppose we view Man as information processor. 7 ’ 6 How 
can we construct models of intelligence that are predictive 
rather than just descriptive? We might build operational 
models, which can exhibit whatever behavior our theory 
called for. To do this, we need to use a general purpose 
symbol manipulator: (/) a way to specify what processing 
gets done when, (ii) a memory in which to store symbol 
structures,** and (Hi) an “engine” that can actually cause 
such processing to occur. The science of Artificial Intelli¬ 
gence (“AI”) sprang into being soon after the invention of 
one such computational engine: the digital computer. In fact, 
AI is sometimes called “Machine Intelligence.” 

The paradigm of AI research 

We in AI have evolved the following paradigm: 

(i) Choose some human cognitive activity (like playing 
chess, proving theorems, understanding spoken Eng¬ 
lish). 

(ii) Develop hypotheses and eventually a theory about 
what kinds of information processing could be taking 
place to produce such ability. 

(iii) Incorporate that theory into a computer program, 
which serves as the model. Make that computer pro¬ 
gram carry out the original activity, and observe how 
well it does.f 

(iv) Experiment with the program, attempting to find out 
where the apparent “intelligence” is really coming 
from. 

Over the last twenty years, we’ve hypothesized and tested 
scores of models, for several different sophisticated tasks. 
There have been many successes (such as the Hearsay 
speech understanding programs), and many failures. Failure 
often stemmed from underestimating how complicated some 
activity is, which human beings are able to do fairly easily 
(for instance, driving a car, or understanding someone who’s 
speaking into a microphone.) We once thought that trans¬ 


* On the other hand, we never capriciously adopt a view with impunity. 

** Symbol processing is not the same as mere data processing. See Reference 
8. It was some years after its discovery that the electronic digital computer 
was perceived as a general symbol manipulator. 

t AI deviates from cognitive psychology at this stage. Psychologists would 
run experiments on people, to see if they really do fit the theory. We in AI 
are more concerned with whether the programs containing the hypothesized 
mechanisms are capable of any sort of “intelligent” behavior—even if it 
differs somewhat from human performance at that task. 


lation by machine could be accomplished by a program 
which simply looked up each word in a Russian/English 
dictionary. Legend has it that “The spirit is willing but the 
flesh is weak ” was translated into Russian as “ The vodka 
is fine but the meat is rotten .” 

We’ve learned a lot from both the successes and the 
failures. Some of these experiments will be described later; 
for now, I just want to present a single, very central result. 

It turns out that we can model a surprising variety of 
cognitive activities (recognizing, problem solving, inventing) 
as a search in which the performer is guided by a large 
collection of informal rules of thumb, which we shall call 
“heuristics” or “heuristic rules.” But what’s really exciting 
is that not only can this single theory (intelligence as heu¬ 
ristic rule guided search) explain the behavior of the brain¬ 
storming math researcher and the wandering Boston visitor, 
but when we go off and build up such models in detail, we 
find that they can all contain many of the same—or at least 
similar—heuristics. If we look at many heuristics (see Figure 
1), we find that each one seems to have its own sphere of 
influence, its own domain of applicability, outside of which 
it is meaningless or useless. There exist many heuristics at 
all these levels; many of the specific ones are just speciali¬ 
zations of one at higher levels (e.g., “ study gene control 
signals across species boundaries ” is a special case of 
“study the scope of biological mechanisms which in turn 
is a special case of “ study the scope of any phenomenon .”) 


HEURISTIC RULE GUIDED SEARCH 

The theory 

There is a theory of intelligence lurking here: 

1. Human cognitive tasks can be cast as searches, as 
explorations wandering toward some goal. 

2. We are guided in these searches by a large collection 
of informal rules of thumb (which we’ve been calling 
heuristics ). 

3. We access relevant heuristics in each situation, and 
follow their advice. 

That’s it.* It sounds plausible; in fact, it sounds trivial. Yet 
the models which incorporate this theory are capable of 
performing many tasks which one would suppose require 
intelligence (organic chemistry problem solving and re¬ 
search, chess playing, composing musical variations, diag¬ 
nosing blood infections, discovering new math concepts and 
conjectures, etc.). (See Figure 2) 


* Notice the conspicuous absence of the word “representation” anywhere in 
the theory. To design and construct a model for this theory would entail 
grappling with representational issues, much as any running instance of an 
abstract algorithm must exist on some particular machine. But the validity 
and power of the theory are independent of representation, just as the validity 
and complexity of an algorithm are independent of which machine it’s imple¬ 
mented on. 
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Program Task 


AM 

DENDRAL 

EURISKO 

Meta-DENDRAL 

M-METHOD 

MOLGEN 

MYCIN 

PECOS 

PROSPECTOR 

PUFF 

RITA 

su/x 

TEIRESIAS 

UT-ITP 


Elementary theory formation and discovery in mathematics 

Enumeration of atom-bond graphs of organic molecules 

Discovery of new mathematics and new heuristics 

Abstraction of new fragmentation rules for Dendral 

Improvization of variations on melodies 

Planning of molecular genetics experiments 

Clinical diagnosis of blood infections 

Expert at coding programs in Lisp 

Evaluating the mineral potential of sites 

Medical expert for pulmonary disorders 

Learns how to run programs at remote Arpanet sites 

Combine many sources’ signals into a global model 

Expert at extracting new Mycin rules from physicians 

Natural deduction proofs of theorems 


Figure 2—Heuristic rule based search program 


Intelligence and information 

The theory contains two important implicit assumptions, 
which are worth displaying explicitly: 

• Man is viewed as a processor of symbolic information. 

• Man exhibits “intelligence” by his performance at var¬ 
ious cognitive tasks. 

They say that information processing has something to do 
with intelligence. Let’s try to justify that. 

Twenty-seven years ago, Alan Turing 9 rejected as mean¬ 
ingless the question “ Can machines think?" He replaced it 
with a game, called the Imitation Game (now commonly 
referred to as the Turing Test). One version of the game 
would go as follows: A human interrogator is placed in an 
isolated room. A teletype exists in the room, and by using 
it he can communicate with a computer and with another 
human, both located in the next room. The interrogator asks 
them each some questions, and then must guess which is 
the human and which is the machine. If we can program a 
machine in such a way that it fools the interrogator into 
making the wrong identification at least 50 percent of the 
time, then we shall say that the machine (as programmed) 
is “intelligent.” Many of you are familiar with this game. 
Now let me introduce you to a slightly different one: 

Thirteen years ago, Keith Gunderson 10 rejected as mean¬ 
ingless the question “ Can rocks think?" He replaced it with 
a game, an Imitation Game. A human interrogator is placed 
in an isolated room. A small hole exists near the bottom of 
the door, through which the interrogator can shove most of 
his foot. On the other side of the door are located a human 
and a rock. Sometimes the human will stomp on the inter¬ 
rogator’s foot, and sometimes the rock will fall on it. The 
interrogator then must guess which one is the human. If we 
can shape a rock in such a way that it fools the interrogator 
at least 50 percent of the time, then we shall say that the 
rock (as shaped) is “intelligent”.* 


* During the oral presentation of this talk at IJCAI-77, the rock test was 
demonstrated onstage, with Allen Newell in the role of the unfortunate in¬ 
terrogator. 


Why does the dialogue-test sound so much less silly, so 
much more indicative of intelligence, than the stomping- 
test? Because unrestricted dialogue is open-ended; it permits 
genuine interrogation of information and information proc¬ 
essing capabilities. The second test doesn’t. If you agree 
that the first test is genuine and the second one bogus, it 
must be because intelligence has something to do with so¬ 
phisticated processing of massive quantities of information. 

Thus it seems that the information processing view of 
Man is an especially good one from which to study intelli¬ 
gence. Before we see what happens when one tries to build 
models based on it, let’s look at six different kinds of situ¬ 
ations requiring intelligence, and see if our theory can handle 
them all. 

Some examples supporting the theory 

Everyday problem solving 

Suppose we decide to plan a trip from CMU to MIT. How 
can we find a good route to take? We will probably obtain 
a detailed road map and begin searching. We have some 
powerful rules of thumb which make our search very short. 
We look for some main highways that will take us most of 
the way, and then do some “fixing up” around the termini. 

The map-makers aid us by drawing the biggest highways 
as very thick red lines. Needless to say, ignoring all but the 
red lines makes the map much simpler. (Thick red lines 
constrain furiously.) The very general heuristic of planning 
has reduced our search dramatically. One could imagine 
even a computer being able to solve it. In the next subsec¬ 
tion, we’ll show that the same kind of analysis can explain 
episodes of brainstorming (in particular, the inventing some 
new kitchen gadget). 

Everyday invention 

Suppose we’re confronted with the following problem: 
we’re fond of eating cheese, but it’s difficult to use a 
knife to cut uniform, thin slices. Let’s assume that we try 
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The thinner the knife, the thinner the slices 

The softer the cheese, the easier it is cut [except: Brie,.. .] 

The slower the slicing, the more uniform the slices 

The longer the knife, the more uniform the slices 

The rounder the whole cheese, the more irregular slices there will be 

etc. 

Figure 3—Cheese-cutting relationships 


to design an improved tool. We are now facing a search in 
an enormous space of possibilities. But we have many in¬ 
formal rules of thumb which may help us: 

1. Sometimes, there will be a good way (perhaps a recent 
invention) to do something more general than what is 
strictly asked for. 

2. Consider what variables affect the success/failure of 
the current (inadequate) technique. Look for motiva¬ 
tion at the extreme cases of the various known rela¬ 
tionships involving those variables. 

3. Look carefully at what is truly wanted; maybe the 
problem can be completely bypassed; at least, perhaps 
it’s over-specified. 

One of them (#2) says to look for motivation at the extreme 
cases of various known relationships. We know several 
things about cutting thin slices (see Figure 3). There seems 
to be some relationship between the thickness of the knife 
and the thinness of the slices we can cut. So we might 
consider as an extreme the thinnest knife possible, just a 
knife edge, and perhaps we thereby invent the wire cheese 
cutter. Or we might look at the extreme case of the relation 
that says that most cheese can be cut thinner if it’s softer, 
moreso than if it’s dry and crumbly. We may ask ourselves 
if we can get the cheese very soft; completely molten; maybe 
then pour it into slice-molds, much like making a thin crepe. 


That’s too messy. If we keep the cheese solid, and use a 
knife, maybe we could just get the cheese directly under the 
knife-edge completely molten; so we’ve invented the “hot- 
knife;” General Electric will probably come out with one 
soon. 

This brainstorming seemed like fun. Incorporating heuris¬ 
tic rules into a computer model which could then tirelessly 
systematically apply them seems like a very promising di¬ 
rection to follow. We’ll follow it in a later section. Bear in 
mind, however, that inventions are rarely made with little 
or no search. In the cheese-cutting case, there might be 
hundreds of somewhat relevant heuristics, and many ways 
of applying each one, and only a few of these combinatorial 
possibilities are even remotely viable. 

On the other hand, both of these other heuristics can be 
used in this situation. The first one is the “inventor’s par¬ 
adox”: sometimes the easiest way to solve a problem is to 
find and solve a more general case, an apparently-harder 
problem. In the cheese-cutting situation, the heuristic might 
direct us to look at various devices which “cut”: a big 
scissors? a paper-cutter? a light-saber? the new whirling- 
nylon-cord grass-cutters? It so happened that recently I 
came across a cheese sheer which operates remarkably like 
a paper-cutter. (See Figure 4). 

Heuristic number three might have us build a better mouse¬ 
trap: go into business selling already sliced cheese; or try 
to devise a “cheese press” that takes the crumbs and 
squeezes or melts them together into shapes that resemble 
cheese slices. “Kraft singles” are just such a product: proc¬ 
essed cheese shaped into slices and then individually 
wrapped. 

Judging “interestingness” 

Let’s put our theory to a most severe test, by asking 
whether it can account for our everyday judgments about 



Figure 4 —Paper-cutter-like cheese slicer 
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MOLECULAR GENETICS EXPERIMENT 

Transform Thy - E. Coli into Thy + E. Coli 

by transferring the relevant gene from the bacteriophage Phi-3-T. 

EXPANDED PLAN 

1. Extract Plii-3-T DNA from bacteriophage Phi-3-T. 

Verify the purity of the DNA by measuring UV absorbtion. 

Check that the molecules are intact, using electron microscope. 

Ensure adequate transforming ability in B. Subtilis. 

2. Add restriction enzymes to break the DNA into fragments. 

Completely digest the DNA, by adding enzyme EcoRi. 

Use electrophoresis to estimate the molecular wts. of the fragments. 

Study, hypothesize, and test until confident of what’s happening. 

Re-do the digestion, but stopping it after partially complete. 

3. Mix the fragments with E. Coli plasmids to construct hybrid plasmids. 

Use EcoRi to cut out psclOl plasmids from E. Coli. 

After mixing these with the fragments, ligate with T4 ligase. 

4. Use these cloned plasmids to transform the E. Coli. 

Grow colonies, selecting for Thy + 

5. Verify the Thy+ character of the transformed E. Coli. 

Use a curing technique to remove the hybrid plasmids from E. Coli. 

Confirm the experiment by reintroducing them into cured bacteria and testing that tl 
observed frequencies of transform stay constant. 

PLAN FOR THE EXPERIMENT 

1. Extract Phi-3-T DNA from bacteriophage Phi-3-T. 

2. Add restriction enzymes to break the DNA into fragments. 

3. Mix the fragments with E. Coli plasmids to construct hybrid plasmids. 

4. Use these cloned plasmids to transform the E. Coli. 

5. Verify the Thy + character of the transformed E. Coli. 

Figure 5—Molecular genetics experiment 


what is and isn’t “interesting.” This would seem to be the 
realm of intuition, emotion, taste, and aesthetics—the an¬ 
tithesis of logical empiricism.* If we look, we find that texts 
on the arts abound with “rules” of composition. They ana¬ 
lyze artistic works in terms of symmetry, recency, uncon¬ 
ventionality, harmony, utility, etc. A typical rule might say: 

IF two apparently disparate parts of the work are suddenly 
recognized as being very closely related, 

THEN that increases the interestingness of both parts, 
and of the whole work as well. 

Charles Dickens’ popularity is due in no small measure to 
his mastery of this heuristic of “coincidence:” the reader is 
always astonished and pleased when two separate characters 
are discovered to be one and the same person. Escher’s art 
is pervaded by just this sort of co-identification (consider 
his print where geese and fish “turn into” one another). 


* If in fact there is a large non-rational component to such judgments, then 
it is not surprising if they lie outside the power of any theory founded on a 
view of Man which ignores that aspect of him. So even a partial success at 
explaining “taste” in terms of heuristic rules would be significant. 


Scientific Problem Solving 

Suppose we’re designing a molecular genetics experiment, 
whose ultimate goal is to transfer a gene from one bacteria 
to another, by using a plasmid (a small, circular DNA mol¬ 
ecule). (See Figure 5) We must design a very long chain of 
steps (perhaps reaching into the thousands), each of the 
form “raise the temperature to x degrees,” “add the follow¬ 
ing restriction enzyme in order to cut the DNA at a certain 
place,” etc. 

It’s a common practice to ignore certain kinds of minor 
steps, and to concentrate on finding a relatively small se¬ 
quence of important intermediate goals. Afterwards, we can 
“expand” each subgoal into a detailed list of small steps. 
(See Figure 5, again) We solved the problem by applying a 
general heuristic—planning—plus several heuristics specific 
to molecular genetics. 

The heuristic used is just the one we used previously to 
find a route from CMU to MIT. There, we chose to ignore 
all minor roads on the map. Here, we chose to ignore the 
minor experimental steps. This repetition is not accidental; 
it illustrates the commonality between everyday problem 
solving and scientific problem solving. Let’s see if this sim- 
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ilarity also occurs between everyday invention and scientific 
invention. 


Scientific invention 

Suppose we’re confronted with the following problem: 
we’re fond of factoring numbers, but it’s difficult to use 
division to cut the numbers up in any uniform, repeatable 
way. E.g., one time, we may factor 12 into 2x6, another 
time into 3x4, and another time it crumbles into even more 
pieces: {1,2,2,3}. We may be content with this situation, or we 
may try to find out more about factoring, to improve our 
“cutting tool”—division—or the way in which we wield it. 

Such an endeavor would involve searching an enormous 
space (looking for a new kind of factoring; looking for new 
facts about factoring). But we have many rules of thumb to 
help us, including the three rules which were displayed ear¬ 
lier. 

One of them (#2) says to look for motivation at the ex¬ 
treme cases of various known relationships. In other words, 
when you’ve got an interesting function/ from A into B, and 
there are some special subsets of B, extreme kinds of B’s, 
then it’s worth defining and studying the inverse images of 
those special parts of B, i.e., the members of A which map 
into those special B' s. (See Figure 6) The function in this 
case is “Divisors-of.” It maps a number into a set of num¬ 
bers. (E.g., Divisors-of [12]={1,2,3,4,6,12}.) An extreme 
case would be when it mapped a number into an extreme 
kind of set, say an extremely small kind of set (e.g., into a 
singleton, an empty set, a doubleton, etc.). 

In other words, this heuristic rule says that it’s worth 
defining and studying the set of numbers with no divisors, 
with just one divisor, with precisely two divisors, and maybe 
the set of numbers with precisely three divisors. Voila ’, 
we’ve invented prime numbers. Notice that we used the 
very same heuristic that led earlier to the invention of the 
wire cheese cutter and the hot-knife. 

Judging “interestingness” 

Even allowing that the “heuristic rule guided search” 
theory could account for the discovery —the definition—of 
prime numbers, could it also explain how a researcher might 
have noticed that they were valuable and interesting? 

Let’s look at how the following four heuristic rules could 
lead to the conclusion that “Primes” is interesting, soon 
after it had first been defined. 

(# 1) IF specializations of concept C have just been cre¬ 
ated, and the current task is to find examples of each 
of them, 

THEN one method is to look over the known ex¬ 
amples of C; they may be examples of some of the 
new specialized concepts as well. 

(#2) IF all examples of a concept C turn out to be ex¬ 
amples of another concept D as well, and C was not 
previously known to be a specialization of D, 



If b is some unusual subset of B, 
Then isolate the subset f~ *(b) of A. 
Figure 6—Go to extremes 


THEN conjecture that C is a specialization of D, 
and raise the “interestingness” value of both con¬ 
cepts. 

(#3) IF all examples of a concept turn out to be in the 
domain of a rarely-applicable function F, 

THEN it’s worth computing all their F-values (their 
images under the function F), and studying that col¬ 
lection of F-values as a separate concept. 

(#4) IF a concept has very few examples (say, less than 

2 ), 

THEN it’s worth trying to find out why, but the 
concept itself is not interesting. 

The first heuristic tells us that IF we’ve just created some 
special kinds of Numbers, and we want to fill in examples 
of them, THEN one quick way to find such examples is to 
look at the known examples of Numbers, say the integers 
from 1 to 1000, and dump each one into whichever new 
specialized set(s) it belongs. If we do this, we get the fol¬ 
lowing empirical results: 

Number with no divisors: (none found) 

Numbers with 1 divisor: 1 

Numbers with 2 divisors: 2, 3, 5, 7, 11, 13, 17, 19, . . . 

Numbers with 3 divisors: 4, 9, 25, 49, 121, 169, 289, 


Heuristic #4 says to rule out any sets which are very tiny; 
that immediately causes us to lose interest in the first two 
kinds of numbers, those with zero or one divisor. None of 
the heuristics have much to say about the next set of num¬ 
bers (with 2 divisors); what about this final set, numbers 
which have precisely 3 divisors? Heuristic #2 says, IF all 
these numbers are also examples of some other special kind 
of number, THEN they’d be interesting. Well, is there any¬ 
thing special about all of them? Most of them seem to be 
odd, . . . , all of them seem to be perfect squares. That’s 
very unexpected and surprising. All these numbers have 
integer square roots; that’s a very rare thing. Heuristic #3 
says that in such a case it’s worth actually computing what 
their square roots are. So let’s take their square roots. Lo 
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and behold! They’re precisely the set of numbers with two 
divisors; i.e., primes. Heuristic #2 explains why we might 
have noticed this, and indicates that it would cause us to 
drastically increase the interestingness values of both Num- 
bers-with-3-divisors and Primes. This heuristic was the same 
one we talked about before, in connection with Dickens and 
Escher: the rule of startling coincidence. 

So we were able to fit this kind of judgmental evaluation 
of “interestingness” into the same theory. 


MODELS 

The examples I’ve shown you so far have been merely 
suggestive. The heuristics which I claimed “explained” var¬ 
ious behaviors might really be just comments on those be¬ 
haviors, ways of rationalizing in hindsight what was actually 
done. I claim that the heuristic rules can be an operational 
part of the theory, that predictive models can be built which 
use them for guidance in solving problems and making dis¬ 
coveries. To that end, I’m going to briefly describe a few 
landmark computer programs that have been written, 
models based on the theory of intelligence as heuristic rule 
guided search. These programs form but a thin sliver of the 
work that has gone on under the label of “Artificial Intelli¬ 
gence.” 

LT and GPS—general problem solving 

Probably one of the earliest AI programs written was the 
Logic Theorist (“LT”, to its friends). 7 It was repeatedly 
given symbolic logic theorems from Principia Mathematica, 
and its task was to find a formal proof of each theorem. LT 
had some given axioms and rules of inference. To search for 
a proof, in a completely exhaustive, systematic manner, 
could literally be an impossible undertaking! But LT had a 
few heuristics which constrained its search: 

“. . . selective principles that enable solutions to be 
found after examining only a relatively tiny subset of 
the set of possibilities. One such principle—illustrated 
by the methods in the Logic Theorist—is to generate 
only elements of the set that are already guaranteed to 
possess at least some of the properties that define a 
solution. Another principle—illustrated by the matching 
process in LT—is to make use of information obtained 
sequentially in the course of generating possible solu¬ 
tions in order to guide the continuing search. A third 
principle—illustrated by the use of similarity tests in 
LT—is to abstract from the detail of the problem 
expressions and to work in terms of the simpler abstrac¬ 
tions.” 

—[Newell and Simon 1972], p. 137. 11 

One of LT’s heuristics was constrained generation: if you 
want an entity satisfying ten specific properties PI . . . P10, 
try to arrange things in such a way that you only have to 
consider entities already guaranteed to satisfy PI . . . P7; 


then just test each one to see if it meets the final three 
criteria. Another principle was to learn from your failures: 
instead of just blindly backtracking, use the nature of your 
failure to help guide you in the future, so you won’t make 
exactly the same mistake again. 

After they learned the power of heuristics from LT, the 
same group of researchers then worked on a program called 
GPS, for General Problem Solver. Its aim was to embed the 
few general heuristics above in a domain-independent form. 
It was hoped that any problem could be represented in the 
GPS formalism, and hence solved by GPS. It was the first 
program to isolate explicitly these general methods: 

1. Means ends analysis: Decide what the most important 
difference is between the current state and the desired 
goal, locate an operator which is known to be relevant 
to reducing that kind of difference, and then try to 
actually apply that operator. With any luck, you’ll be 
one step closer to the goal. 

2. Setting up subgoals, equivalent to the idea of problem 
reduction, and to divide and conquer strategies. 

3. Planning in an abstraction space. 

Two decades of experimenting with GPS have shown that, 
yes, a large array of problems can be cast in terms that GPS 
can deal with (states, operators, difference tables), but no, 
GPS’s few general heuristics just aren’t powerful enough to 
solve problems as difficult as can be tackled by humans. 

DENDRAL—scientific problem solving 

The next giant step along the line of development we’re 
following was taken by Ed Feigenbaum 24 and Joshua Led- 
erberg 22 at Stanford, and independently by Joel Moses 23 here 
at MIT. They recognized what was lacking: expertise. After 
all, humans have to train in specialized fields for quite a 
while before they are able to solve any hard problems in 
them. This phenomenon might not just be a reflection of 
human brain frailties, it might indicate a necessary require¬ 
ment for intelligent problem solving in a complex knowl¬ 
edge-rich domain. 

In 1965 they conceived a new kind of computer program, 
and in the process a whole new approach for AI: they were 
willing to commit their programs to working on a very spe¬ 
cific kind of problem—in Moses’ case, symbolic integration; 
in Lederberg, Feigenbaum, and Buchanan’s 25 case, the enu¬ 
meration of atom-bond graphs of organic molecules (based 
on analysis of mass spectrograph data and nuclear mass 
resonance data). For instance, you type in “Ci 0 H 22 ,” and 
their program types back a list of all 71 structural isomers 
of decane. They built their program, DENDRAL, around a 
body of heuristics—not just a few, but a few dozen heuristic 
rules. Some of them were as general as the rules that GPS 
had, but most of them were specific rules of thumb which 
they extracted from chemists (e.g., rules for recognizing 
imbedded subgraphs which are impossible; knowledge of 
aggregates of radicals which co-occur often; heuristic esti¬ 
mation of the worth of pursuing a particular subproblem; 
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Figure 7—Generality vs. power 

rules for recognizing current subproblems which have been 
worked on before). 

The success of this research (measured partly by the large 
number of articles which have been published in chemical 
journals) confirmed Feigenbaum’s hypothesis about the 
tradeoff between generality and power. (See Figure 7) Heu¬ 
ristics are distributed along the pictured curve (and below 
it); the more general they are, the weaker they must be. 
Experiments with Dendral have demonstrated the impor¬ 
tance of using both general heuristics and task-specific heu¬ 
ristics for guidance. 

AM—scientific invention 

We now jump to some quite recent research by the 
speaker, which involved collecting hundreds of heuristics, 
to do elementary mathematics research, open-ended discov¬ 
ery. This is scientific invention, as compared to Dendral, 
which performs scientific problem solving .* 

Math discovery as heuristic rule-guided search 

The task which this program, called AM, performs is the 
discovery of new mathematics concepts and relationships 
between them. The simple paradigm it follows for this task 
is the one specified by our theory: 

1. Open-ended math research is viewed as a search, an 
exploration in a space of partially-developed con¬ 
cepts.** 

2. AM is guided in this exploration by a collection of a 
few hundred heuristic rules. They are rules of varied 
generality which guide it to define and study the most 
plausible thing next. 

3. In each situation, AM quickly accesses relevant heu¬ 
ristic rules, and then follows (obeys) them. 

For example, AM discovered “Multiplication” in four 


* This is not the “inductive vs. deductive” issue: both Dendral and author’s 

program (AM) perform quite inductive tasks. The difference is that Dendral 

is given specific problems to solve (specific mass spectra to identify), whereas 

AM’s activities are open-ended research, with no particular goal specified. 

** The goal of this search is ill-defined: to maximize the interestingness value 
of the concepts which are available for AM to develop further. 


separate ways. This coincidence raised its interestingness 
value to quite a high level. One of AM’s heuristic rules said 
“If f is an interesting operation, Then look at its inverse. 1 2 3 * * * * * 9 ’ 
This rule was relevant (with /= Multiplication) and was ac¬ 
tually followed in this context. It directed AM to define and 
study the relation “divisors-of.” Another heuristic rule 
which later was obeyed (see Figure 8) said to examine the 
inverse image of extrema under interesting functions. We’ve 
seen this rule before, but Figure 8 indicates more accurately 
how it appeared in the AM computer program. Just as we 
saw before, AM was driven by this rule to define prime 
numbers. 

The same heuristic also directed AM to study not only 
various kinds of numbers having very few divisors, but also 
classes of numbers with very many divisors; such “maxi- 
mally-divisible” numbers were also found to be interesting 
(by AM, by the author, and by professional mathemati¬ 
cians). In fact, a couple minor new results in number theory 
were developed regarding them. Still another time, AM was 
guided by the same heuristic to define and study the pairs 
of sets whose intersection is empty; i.e., this rule also led 
to the definition of the relation “Disjointness.” This multiple 
usage of the heuristics is strong evidence that they are truly 
general, t 

Representation of mathematical knowledge 

What exactly does it mean for AM to “have the notion 
of’ a concept? It means that AM represents that concept 
somehow in the computer, that there is a data structure of 
some kind which is meant to correspond to, and contain 
information about, that concept. While the entire issue of 
how to represent knowledge has been de-emphasized in this 
talk, it is helpful to glance at how some typical concepts 
looked. (See Figure 9) 

The representation of a concept is as a collection of facets, 
or slots, (e.g.. Definitions, Conjectures, Wortht) each of 
which can have some associated value, or entry. 

Flow of Control 

AM is initially given a collection of 115 core concepts, 
(see Figure 10) each with only a few facets (slots) filled in. 
AM repeatedly chooses some facet of some concept, and 
tries to fill in some entries for that particular slot. Thus a 
“job” for AM is simply to engage in a mini-research project, 
to commit a simple act of discovery. Its overall task—to 
discover interesting concepts and conjectures—is accom- 


t If a supposedly general rule were only involved in one major discovery, 
there would always be the question of whether the rule is merely a subtle 
encoding of that discovery. 

t Unlike the other numbers appearing as part of the pictured ‘Primes’ con¬ 
cept, the number “800” is merely an estimate, an ad hoc value for the overall 
worth of this concept. It would be preferable to replace this by a mass of 
symbolic reasons, by a plausible justification for why this concept was and 
wasn’t useful, interesting, etc. But in lieu of the ability to reason about, 
create, manipulate, and otherwise use such knowledge, we opt for a simple 
scalar, a numeric rating which in effect crudely summarizes these reasons. 
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In English and Math notation: 

IF f: A -*> B is an interesting function, 

and there’s some interesting or extremal subset X of B. 
THEN it’s worth defining and naming and studying f - '(X). 


Roughly as it appeared in the AM program: 

(IF 

(CURRENT-TASK DEALT-WITH “F”) 

AND (F IS-A FUNCTION) 

AND ((WORTH-OF F) IS-GREATER-THAN 600) 

AND (THERE-ARE-KNOWN-ENTRIES-FOR (BOUNDARY-OF (RANGE-OF F))) 
THEN 

(FOR-EACH X IN (BOUNDARY-OF (RANGE-OF F)) 
(CREATE-A-NEW-CONCEPT 
(WHOSE NAME IS “F-INVERSE-OF-X”) 

(WHOSE WORTH • IS (WORTH-OF F)x (WORTH-OF X)) 

(WHOSE DEFINITION IS (LAMBDA (z) (F (z) IS-A X))) 

(WHOSE GENERALIZATION IS (DOMAIN-OF F)) 

(WHOSE ORIGIN IS “Inverse image of extrema”) 

(WHOSE INTEREST IS “(TIES-TO F) (TIES-TO F-INVERSE)”)))) 

Figure 8—Go to extremes, again 


plished as a composition of hundreds of these repeated at¬ 
tempts at little discoveries. 

To decide which small job to work on next, AM maintains 
an agenda of jobs, a global queue ordered by priority. (See 
Figure 11) A typical job is “Fill-in examples of Primes.” 
The agenda may contain hundreds of such jobs. AM re¬ 
peatedly selects the top-rated job from the agenda and tries 
to find relevant rules (i.e., heuristics which, when obeyed, 
bring AM closer to satisfying that job). Potential relevance 


is determined a priori by where the rule is stored. Each of 
AM’s 250 heuristic rules is tacked onto the most general (or 
abstract) concept** C to which it’s relevant and meaningful. 
The relevance of these heuristic rules is presumed to be 
inherited by all C’s specializations. A method which can 


** It was very important to place each heuristic on as general a concept as 
possible, to give it a large scope of applicability, to bring it close to the 
generality versus power curve. 


NAME(s): Set, Non-proper Gass, Collection, Finite set 
DEFINITIONS: 

RECURSIVE: A (S) [S = {} or Set. Definition 
(Remove(Any-member(S),S))] 

RECURSIVE QUICK: A (S) [S = {} or Set. Definition (CDR(S))] 
QUICK: A (S) [Match S with {...}] 

SPECIALIZATIONS: Nonempty-set, Set-of-sets, Set-of-numbers 
BOUNDARY: Empty-set, Singleton, Doubleton, Tripleton 
GENERALIZATIONS: Unordered-Structure, Collection, 
Structure-with-no-multiple-elements-allowed 
IS-A: Kind-of-structure 
EXAMPLES: 

TYPICAL: {{}}, {A}, {A,B}, {3} 

BARELY: {}, {A, B, {C, {{{A, C, (3,3,9), < 4,{B |.A>}! }}} 
NOT-QUITE: (A,A},(), (B,A) 

FOIBLE: <4,1,A,1> 

OONJEC’s: All unordered-structures are sets. 

INTUITIONS: Geometric: Venn diagram. 

ANALOGIES: {set, set operations} = {list, list operations! 
WORTH: 600 [on a scale of 0-1000] 

VIEW: 

PREDICATE: A (P) {xeDomain(P) | P(x) j 
STRUCTURE: A (S) Enclose-in-braces 

(Sort(Remove-mul t iple-elements(S))) 
IN-DOMAIN-OF: Union, Intersection, Set-difference, Subset, 
Member, Cartesian-prod, Set-equality 
IN-RANGE-OF: Union, Intersect, Set-difference, Satisfying 


NAME(s): Prime Numbers, Primes, Numbers-with-2-Divisors 
DEFINITIONS: 

ORIGIN: Divisors-of(x) is-a Doubleton 
PRED.-CALCULUS: Prime(x) m (Vz)(z|x -> z =1 XOR z = x) 
ITERATIVE: (for x > 1): For i from 2 to x-1, ~i(i|x) 
EXAMPLES: 2, 3, 5, 7, 11, 13, 17 
BOUNDARY: 2, 3 
BOUNDARY-FAILURES: 0, 1 
FAILURES: 12 

GENERALIZATIONS: Numbers, Numbers with an even no. of 
divisors, Numbers with a prime no. of divisors 
SPECIALIZATIONS: Prime pairs, Prime uniquely-addables 
CONJECS: Unique factorization, Goldbach’s conjecture 
ANALOGIES: Maximally-divisible numbers are converse 
extremes of Divisors-of 

INTEREST: Conjee’s tying Primes to Times, to Divisors-of, and 
to other closely related operations 
WORTH: 800 


Figure 9—Two typical math concepts 
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Anything 



Relation 


Logical-rein 

Constant-pred Equality-pred 


Struc-of-strucs 


Const-T Const-F Obj-equal 


Coalescing 
Inverted-operation 
Canonization 
Composition 
Restricted-operation 
All-but-first, All-but-last 
First-element, Last-element, 

Project, Repeat, Restrict, Reverse-ordered-pair 
Identity, Invert-op, Parallel-join, Parallel-replace 
Set-delete, Set-difference, Set-insert, Set-intersect 
Set-union, Set-equality, Bag-delete, bag-union,... 


Mult-eles Non-mult Ord Unordered 

\/ 

Osets 




Ord-pairs 


The diagram above represents the “topmost” concepts which AM had initially, shown 
connected via Specialization links (\) and Examples links (|||). 

Figure 10—Initial concepts 


(.Fill in Generalizations facet of “Equality concept ”) 

Priority = 700 

Reasons = ((No known generalizations of Equality) 

(Equality is rarely satisfied) 

(Focus of attention: recently worked on Equality)) 
(Check Conjees facet of “Equality” concept) 

Priority = 500 

Reasons *= ((Worth of Equality has recently risen) 

(Conjees involving Equality have just been found) 
(Focus of attention: recently worked on Equality)) 
(Fill in Examples facet of “Primes concept ”) 

Priority — 400 

Reasons = ((No known exs of primes yet)) 

(Check Examples facet of “Set-union” concept) 

Priority = 350 

Reasons = (Many examples of Set-union recently added) 
(Worth of Set-union has recently risen)) 

(Check Domain/range facet of “Square-root” concept) 

Priority = 300 

Reasons = ((Square-root was unable to return a value for 8) 
(Square-root was unable to return a value for 3) 
(Should check the domain/range entries for any 
operation created as an inverse) 

(Fill in new Algorithms for Compose concept) 

Priority = 200 

Reasons = ((Empirical: taking too long)) 

Figure 11—Agenda of jobs 
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generalize any operation can certainly generalize any com¬ 
position; any heuristic that knows how to find examples of 
any concept can find examples of “ Numbers-with-3-divi- 
sors.” And so on. In other words, the aggregate of the 
Generalization/Specialization relationships among the con¬ 
cepts induces a similar graph structure upon the set of heu¬ 
ristic rules. (See Figure 12 and also Figure 1, again) This 
“inheritability property” permits potentially relevant rules 
to be located efficiently. 

The IF-part of each potentially relevant rule is evaluated 
to determine whether the rule is truly relevant. If so, it’s 
obeyed. At that time, it can direct three kinds of actions or 
effects: 

(i) Facets of some concepts can get filled in (e.g., ex¬ 
amples of primes may actually be found and tacked 
onto the “Examples” facet of the “Primes” concept). 

(ii) New concepts may be created (e.g., the concept 
“numbers which are uniquely representable as the 
sum of two primes” may be explicitly deemed worth 
studying). 

(Hi) New jobs may be added to the agenda (e.g., the 
current activity may suggest that the following job is 
worth considering: Find new generalizations of the 
concept of prime numbers”). 


Behavior of this Rule System 

AM began its investigations with scanty knowledge of one 
hundred elementary concepts of finite set theory. Most of 
the obvious set-theoretic concepts and relationships were 
quickly found (e.g., de Morgan’s laws; singletons), but no 
sophisticated set theory was ever done (e.g., diagonaliza- 
tion). Rather, AM discovered natural numbers, rated them 
highly, and went off exploring elementary number theory. 
(See Figure 13) AM discovered scores of well-known con¬ 
cepts (some of them in novel ways,* some of them in several 
ways**), and a roughly equal number of “losers” (most of 
which were quickly recognized as such by AM). All this 
occurred in two cpu hours (see Figure 14). 

The AM project has demonstrated that that open-ended 
scientific theory formation (the defining and exploring of 
new concepts and relationships) could be mechanized, could 


* E.g., AM restricted the operation ‘Addition’ to prime numbers; i.e., it 
asked “For which primes p,q,r is it true that p+q=rl" In such a case, p or 
q must be “2',” and the other two primes must be a prime pair. 

** E.g., multiplication was discovered in four separate ways: as repeated 
addition, as an analogue to the Cartesian product of sets, as the cardinality 
of the union of the powersets of two sets, and as the total number of symbols 
one gets by replacing (in parallel) each element of bag AT by a complete copy 
of bag Y. 


ANYTHING 

INTEREST: The user recently mentioned C. 

INTEREST: C is the result of F(x), where F and x are interesting. 

ANY-CONCEPT 

FILL-IN: Find the results of any operation whose range is C. 
INTEREST: The Worth rating of C keeps fluctuating wildly. 

CHECK: For any x, x is a C iff it satisfies the Definition of C. 

ACTIVITY 

FILL-IN: Run the algorithm for C on random members of its domain. 
INTEREST: All the components of C’s domain are identical. 

OPERATION 

FILL-IN: For any identity i of C, for any j, include ‘C(i, j) = j\ 
INTEREST: The range of C is also one of C’s domain components. 


COMPOSITION 

INTEREST: CoC is nontrivial. 

INTEREST: F’s conjees hold for C = FoG. 
CHECK: Dom(F) f) Rang(G) is nontrivial. 


DOMAIN - RANGE-OP 

FILL-IN: Include any fixed-points of C. 
INTEREST: C(x, C(y, z)) = C(C(x, y), z). 
CHECK: Ran(C) is not a proper subset of 
Dom(C). : 



UNION • COMPLEMENT 

Figure 12—Induced structuring of the heuristics 
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Sets with less than 2 elements (singletons and empty sets). 

Sets with no atomic elements (nests of braces). 

Singleton sets. 

Bags containing (multiple occurrences of) just one kind of element 
Superset (contains). 

Doubleton bags and sets. 

Set-membership. 

Disjoint bags. 

Subset. 

Disjoint sets. 

Singleton osets. 

Same-length (same number of elements). 

Same number of left parentheses, plus identical leftmost atoms. 
Count (find the number of elements of a given structure). 
Numbers (unary representation). 

Add. 

Minimum. 

SUB1 (A(x)x-l). 

Insert x into a given Bag-of-T’s (almost ADDJ, but not quite). 
Subtract (except: if x < y, then the result of x —y will be zero 71 ). 
Less than or equal to. 

Times. 

Union of a bag of structures. 

& (the ampersand represents the creation of several real losers.) 
Compose a given operation F with itself (form F.F). 

Insert structure S into itself. 

Try to delete structure S from itself (a loser). 

Double (add ‘x’ to itself). 

Subtract ‘x’ from itself (as an operation, this is a real zero 72 ). 
Square (TIMES(x,x)). 

Union structure S with itself. 

Coalesced-replace2: replace each element s of S by F(s,s). 
Coalesced-join2: append together F(s,s), for each member seS. 
Coa-repeat2: create a new op which takes a struc S, op F, and 
repeats F(s,t,S) all along 
Compose three operations: A(F,G,H) F.(G.H). 

Compose three operations: A(F,G,H) (F. G). H. 

& (lots of losing compositions created, e.g. Self-insert.Set-union.) 
ADD _1 (x): all ways of representing x as the sum of a bunch of 
nonzero numbers. 


G . H, s.t. H(G(H(x))) is always defined (wherever H is), and G and H 
are interesting. 

Insert. Delete. 

Delete. Insert. 

Size ADD '. (A (n) The number of ways to partition n) 

Cubing 

& 

Exponentiation. 

Halving (in natural numbers only; thus Halving(15)—7). 

Even numbers. 

Integer square-root. 

Perfect squares. 

Divisors-of. 

Numbers-with-O-divisors. 

Numbers-with-1 -divisor. 

Primes (Numbers-with-2-divisors). 

Squares of primes (Numbers-with-3-divisors). 

Squares of squares of primes. 

Square-roots of primes (a loser). 

TIMES - 1 (x): all ways of representing x as the product of a bunch of 
numbers (•■!). 

All ways of representing x as the product of just one number 
(a trivial notion). 

All ways of representing x as the product of primes. 

All ways of representing x as the sum of primes. 

All ways of representing x as the sum of two primes. 

Numbers uniquely representable as the sum of two primes. 

Products of squares. 

Multiplication by 1. 

Multiplication by 0. 

Multiplication by 2. 

Addition of 0. 

Addition of 1. 

Addition of 2. 

Product of even numbers. 

Sum of squares. 

Sum of even numbers. 

& (losers: various compositions of 3 operations.) 

Pairs of perfect squares w hose sum is also a perfect square 
(x 2 +y 2 —z 2 ). 

Prime pairs (p,p 2 are prime). 


Figure 13—Concepts discovered 


be modelled as heuristic rule guided search, using a few 
hundred heuristics for guidance. This is a significant verifi¬ 
cation of the theory of intelligence presented earlier. 

AM had some difficulties. As it ran longer and longer, the 
concepts it defined were further and further from the prim¬ 
itives it began with; while the general set-theoretic heuris- 

Machine: SUMEX, PDP-10, KI-10 uniprocessor 

Language: INTERLISP, January 1975 release 

Number of small support functions for AM: 200 

Number of heuristics: 250 

Number of Initial concepts: 115 

Number of Final concepts: 300 

Average Initial concept: 8 facets 

Average Final concept: 11 facets 

Average size of a concept: 700 list cells 

Average size of a concept: 1 typed page 

Total CPU Time: 1 hour 

Speedup factor if user advises AM: 3 

Total number of jobs executed: 200 

Average time allocated per task: 30 secs. 

Average time actually used by a task: 18 secs. 

Number of winning concepts: 25 (estimated) 

Number of acceptable concepts: 100 (est.) 

Number of losing concepts: 60 (est.) 

Figure 14—Performance statistics 


tics were technically valid for dealing with primes and arith¬ 
metic, they were simply too general, too weak to guide 
effectively. The key deficiency was AM’s inability to create 
and modify new specific heuristics. 


The next step: EURISKO 

These findings came to light only because the AM program 
existed, was run, and was experimented with. Recall that 
the final step in the AI paradigm is to experiment with the 
computational model. In this case, what we’ve learned from 
AM is guiding our current efforts at writing a program which 
discovers new task-dependent heuristics as well as new con¬ 
cepts. Our new program, EURISKO, is quite ambitious, 
because even scientists are very poor at formulating—or 
even recognizing—new heuristics. 

We asked ourselves how new heuristics get discovered; 
we took the same kind of heuristically-guided approach to 
the problem that AM would have. By using general rules, 
we were led to ponder whether heuristics mightn’t get dis¬ 
covered and reasoned about in much the same way that 
math concepts do (i.e., by hindsight, by manipulating exist¬ 
ing ones, and so on). We think they might. 

So EURISKO’s method for discovering and developing 
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NAME(s): Generalize-rare-predicate 
STATEMENT 

English: If a predicate is rarely True, create a generalization of it 
IF-Current-task-deals-with: a predicate P 
AND-IF: P returns “True” less than one tenth of the time 
THEN-Conjecture: 

Cl: P is a little less interesting than expected. 

C2: Generalizations of P may be more interesting than P. 
Then-add-a-new-task: 

“Fillin Generalizations of P” because C2 
Theh-modify-facet: 

Reduce the Worth of P by 10%, because t'l 
SPECIALIZATIONS: Generalize-rare-set-predicate 
BOUNDARY: Enlarge-domain-of-predicate 
GENERALIZATIONS: Modify-predicate, Generalize-concept 
JUSTIFYING GENERALIZATION: Hill-climbing 
1S-A: Heuristic 
EXAMPLES: 

GOOD: Generalize “scl-equalily” into “same-length” 

BAD: Generalize “set-equality” into "same-first-elemenl" 

CONJEC’s: Special cases of this rule are MUCH more powerful than it is. 
ANALOGIES: Weaken-an-ovcrconstraincd-problem 
WORTH: 600 [on a scale of 0-1000] 

VIEW: 

ENLARGE-STRUCTURE: let thestructuie be {x | P(x); 

ORIGIN: A specialization of “Modify-predicate”, via empir. induction 

HISTORY 

Summary of examples: l good, 1 bad example known; average -r0.4 

Summary of conjectures proposed: 3 correct, 1 unclear 

Summary of tasks added to agenda: 2 added, 2 tried, 2 succeeded, -f 0.8 

Summary of facets modified: 2 Worth lowerings, 1 verified 

CPU Time used: Average 9.4 epu seconds 

Space used: Average 250 list cells 

Figure 15—Heuristic represented as a concept 


heuristics is therefore simply to not distinguish between 
concepts and heuristics; i.e., each heuristic is represented 
internally as a full-fledged concept. (See Figure 15)* Any 
heuristic which, say, can advise when it’s time to generalize 
any concept, can also automatically tell when it’s time to 
generalize any heuristic. Any method for creating a new 
concept out of old ones can be used to create new heuristics 
out of old ones. Evaluating the new heuristics is done just 
like evaluating any new concepts: by observing them in 
action, by gathering empirical data about them. Naturally, 
there are some special heuristics which are meaningful only 
for reasoning about other heuristics, just as there are some 
special heuristics that apply only to ligating plasmids. 

We’ve taken the same AM-like approach to solving most 
of the problems that beset AM. 

Other heuristic rule guided expert programs 

There are several programs like AM in existence, knowl¬ 
edge based expert programs which perform under the guid¬ 
ance of a large collection of heuristic rules. 

The MYCIN program 12 contains a couple hundred judg- 


* Notice its similarity to a typical math concept. 


mental rules which were extracted from physicians, and 
it uses them to make diagnoses of various blood infec¬ 
tions. 

MOLGEN 13 (still under construction) attacks the molec¬ 
ular genetics experiment-planning problem discussed ear¬ 
lier. 

The PROSPECTOR program 14 performs geological anal¬ 
ysis of on-site data relayed to the computer by a human 
expert. It aids him in the evaluation of the mineral poten¬ 
tial of exploration sites. Prospector’s rules were gleaned 
from geologists, much as MYCIN’s were from physicians. 

These programs perform complex problem solving. A few 
programs exist which, like AM, use heuristic rules to guide 
them in scientific and other invention tasks. 

META-DENDRAL 15 is a theory formation program which 
looks over mass spectra and their correct identifications, 
and then abstracts that data into new fragmentation rules, 
which are then usable by the Dendral program, as if they 
had been extracted from a human expert. 

The M-Method program 16 for improvising variations on a 
given melody is based around a body of musicology rules. 

Several other such programs are listed in Figure 2. 
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CONCLUSIONS 

The main points to remember 

• We viewed Man as a processor of symbolic information. 
Keeping in mind that this is but one narrow view, we 
found it a useful vantage point from which to study the 
phenomenon of intelligence. 

• Many cognitive tasks can be cast as explorations 
through more or less enormous* search spaces. “Intel¬ 
ligence” is the ability to zero in effectively on a solu¬ 
tion, despite the apparent size of the search space. 

• We theorize that humans possess thousands of informal 
rules of thumb, of varying levels of generality and 
power, which they use to guide them in these searches. 
These rules are the prime repository for domain-specific 
knowledge (i.e., for expertise). 

—We have seen the relationships among knowledge, 
search, and intelligence. 

Search is ubiquitous. 

Knowledge reduces search. 

Using knowledge is intelligent. 

—It is therefore meaningless to debate whether the 
source of intelligence lies in “search” or in “knowl¬ 
edge:” both are prerequisites. They interact syner¬ 
getically: each would be almost powerless without 
the other. Search by itself leads to small programs 
with impossible running times. Knowledge by itself 
just sits on library shelves and gets dusty. 

—The theory of intelligence as heuristic rule guided 
search contains the power of additivity: many small 
pieces of “local knowledge” combine to produce so¬ 
phisticated “global effects,” with a consequent ease 
of introducing new pieces of knowledge. 

—This methodology contains the power** afforded by 
coarse granularity: a rule can express any arbitrarily 
complex heuristic, can embody a meaningful chunk 
of domain-specific knowledge. In particular, it can 
have some significance to a human expert. Thus each 
model of the theory will be self-explaining: 17 it need 
only keep the user posted of each rule firing. Also, 
new heuristic rules can be added by a human expert 
at a level which seems “natural” to him. 18 

• Several example scenarios have been presented, to 
show that this theory can account for everyday and 
scientific problem solving, invention, and judging the 
“interestingness” of newly synthesized inventions. 
There is a whole lattice of increasingly more specialized 
heuristics. Some of them, at the very top, are useful in 
all six types of situations. 

• Three kinds of models, based on this theory, were ex¬ 
amined (LT/GPS, Dendral, and AM). The efforts are 
separated in time by periods of about ten years (1957, 


* The less structured the domain, the larger the search space. One extreme 
of this “tradeoff” is an algorithm, which is a highly “compiled” search. 

** Promised by general “heuristic search,” 19 and by some rule-based system 
architectures, 20 but not by “pure” production systems. 21 


1967, 1977) and factors of about ten in the number of 
heuristics they utilized (3, 30, 300). All three serve to 
support the theory of intelligence we’ve been discuss¬ 
ing. 

• We saw in particular how AM is able to use its heuris¬ 
tics to guide attention (add and modify jobs on its 
agenda), to synthesize new concepts, to explore them 
(fill in their facets), and to evaluate their interesting¬ 
ness. This demonstrated that the general heuristic rules 
are not merely a descriptive commentary, they are an 
operational part of the theory. AM and its predecessors 
are not merely lucky accidents. 

The goals of AI 

Toward a science of science 

In the Middle Ages, only a few European artists incor¬ 
porated linear perspective into their paintings. The tech¬ 
niques were never communicated explicitly; rather, the 
would-be artist had to serve as an apprentice to a master. 
Gradually, by example, the student learned the necessary 
skills. Some time between Giotto and Da Vinci, the artists 
wrote down, explicitly, the knowledge they were really 
using, to create perspective. Once they did that, anyone 
could learn how to create such drawings. In place of mystery 
was understanding. 

There is a similar mystique today, about how doctors can 
diagnose disease, how mathematicians can discover new 
theorems, how scientists in general are able to carry out 
productive research. The doing of science today is not unlike 
the guilds of the Middle Ages. We scientists rarely try to 
introspect on our own creativity, to explicate our skills. 

One very significant reason for doing Artificial Intelli¬ 
gence research is that it starts the metamorphoses of the 
nature of scientific fields, from incomprehensible guilds into 
well-understood technical crafts. It leads to a science of 
science. AI programs make explicit, in a very usable form, 
the knowledge necessary to perform some complicated ac¬ 
tivity, thereby de-mystifying it. 

Mental enhancement 

A concrete understanding of intelligence, as might be 
achieved by the above “Science of Science”, enables at¬ 
tempts to synthesize it. What good would a synthetic intel¬ 
lect be? 

The ultimate material benefit to be derived from Science 
is the enhancement of (some aspect of) Man. Medicine views 
him as an organism, and is able to provide biological en¬ 
hancement (pharmacological agents and techniques which 
improve his resistance to disease, his longevity, etc.). Engi¬ 
neering views him as a dynamic actor, and is able to provide 
physical enhancement (devices and techniques which enable 
better locomotion than legs, better communication that 
shouting, etc.). AI views Man as a symbol processor, and 
provides mental enhancement: programs and techniques 
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which augment his ability to reason: automation of routine 
tasks (bookkeeping, housekeeping*), aids to artists and sci¬ 
entists (partial automation of the “hack” side of composing, 
proving, diagnosing, exploring, programming), an avenue 
leading to the understanding (and appreciation of the so¬ 
phistication) of his own intelligence, and perhaps—in the far 
distant future—even finding his way around the streets of 
Boston. 
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PANEL OVERVIEW—Saul Amarel 

The papers in this session will focus on applications of 
Artifical Intelligence in the development of knowledge-based 
systems in medical decision-making situations, in scientific 
inquiry, and in mineral exploration. 

Much of the work which will be reported in this session 
is being done within the AIM community. [AIM stands for 
Artificial Intelligence in Medicine.] The AIM project is a 
national resource sharing activity supported by the biotech¬ 
nology resources program (BRP) of DRR, NIH, whose prin¬ 
cipal objective is to promote applications of AI to medicine 
and the life sciences. The main focus of AIM activity is at 
the Stanford University SUMEX-AIM project, which pro¬ 
vides computer shared resources to the AIM community via 
national networks. The Rutgers research resource on com¬ 
puters in biomedicine 1 is one of the (BRP-supported) proj¬ 
ects in the AIM community. Included in its functions is the 
organization of AIM workshops. 2 The objective of the work¬ 
shops is to strengthen scientific interactions within the na¬ 
tional AIM community and to disseminate Al-based meth¬ 
odologies, tools, and specific systems that are relevant to 


AIM. The next AIM workshop (this is the fourth) will be 
held at Rutgers on June 25-28, 1978. All the participants in 
this session are expected to attend the Rutgers workshop 
and to discuss in depth various aspects of their research on 
knowledge-based systems. The present session will provide 
an opportunity to bring summary accounts of this research 
to a wide national audience of computer professionals; a 
similar session was held last year at the Fifth International 
Joint Conference on Artificial Intelligence. 3 

Experience with work in AI applications to date shows 
the following: 

(a) The problem of acquiring specific knowledge in a do¬ 
main, managing it in an AI system, modifying it, and 
using it appropriately is fundamental. The approach to 
most designs is incremental and responsive to the fact 
that the knowledge base in the domain is not station¬ 
ary. Initially, a relatively low-performance AI system 
is created to provide the basis for subsequent stages 
of knowledge acquisition and improvement, which 
eventually leads to high performance, and expert sys¬ 
tems. 

(b) Work on applications requires very close collabora¬ 
tion between AI experts and experts in the problem 
domain. Furthermore, special technical support facil¬ 
ities (e.g., computer networks) can play a significant 
role in establishing these collaborations and in sus¬ 
taining their effectiveness. This is a key point which 
has important implications for organizational and 
shared resource aspects of applied AI projects. 

(c) The development of an expert system within a rea¬ 
sonable time span requires more powerful technolo¬ 
gies than those in use today—especially when the 
knowledge bases will grow from the present 10**2 to 
10**3 “facts” to more realistic situations with 10**4 
to 10**5 “facts”. So far, system development times 
(from conception to expert level in a research envi¬ 
ronment) have been 4 to 8 years. To reduce this time 
span, or to keep it from growing too much as knowl¬ 
edge bases grow, we need more effective methods of 
knowledge acquisition and organization and more 
powerful program design environments. Related to 
this, we need better techniques for interfacing AI pro¬ 
grams with experts and users. At a more basic level, 
we need better schemes for coordinating multiple 
knowledge bases and for handling inconsistent and/or 
uncertain information. 
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Experience of several years in the development of clinical 
decision-making “expert” systems at MIT to be discussed 
by Peter Szolovits illustrates the ways in which these issues 
are encountered in specific AI application projects. Another 
important conclusion obtained from the MIT experience is 
that the choice of medical tasks to be approached via AI 
methodologies/techniques should be made after a careful 
assessment of the nature and complexity of the task. Only 
tasks requiring complex knowledge bases and difficult rea¬ 
soning processes should be approached via AI methods; in 
many situations simpler computer-based methods may be 
more appropriate. 

One of the key processes in medical reasoning and in 
scientific inquiry is the interpretation of empirical data in 
the light of a given body of theoretical knowledge. Much of 
the work presented in this session is concerned with these 
processes. Typically, a problem of interpretation involves 
reasoning about an individual case. Given evidence (data) 
about the case and a theory within which the evidence is to 
be understood/explained, find a hypothesis which explains 
the case on the basis of the theory. While there has been 
considerable progress in the development of strategies for 
solving interpretation problems, there still remain several 
open problems: under what conditions the interpretation 
process should be controlled by the specific, “low level” 
features of the case under consideration or by possible “high 
level” hypotheses and by expectations derived from these 
hypotheses; how to best represent and keep track of infor¬ 
mation about a special case, of general knowledge in terms 
of which the case is interpreted, and of alternative interpre¬ 
tations of the case as the process evolves with time. 

Some of these issues will be discussed in depth by Harry 
Pople in the context of his research (in collaboration with 
Dr. Jack Myers) on diagnostic problem solving in internal 
medicine. These are also important issues in (I) the work of 
John Kunz, Larry Fagan and their collaborators in the do¬ 
main of pulmonary disorders, (II) the work of Richard Duda, 
Peter Hart and their collaborators in the domain of mineral 
explorations, and (III) the work of Sridharan, which is ori¬ 
ented to the development of an AI system for assisting in 
the formulation of psychological theories of action interpre¬ 
tation. 

Processes of planning are becoming another focus of in¬ 
terest in AI applications to natural systems. The synthesis 
of plans has been a subject of research since the early days 
of AI. In this session, the synthesis of treatment plans in a 
medical context (consultation in glaucoma) will be discussed 
in the paper by Casimir Kulikowski and Sholom Weiss. AI 
approaches to the development of plans in the context of 
scientific experimentation (in molecular genetics) will be 
discussed in the paper by Mark Stefik. Research on the 
analysis/interpretation of plans is of a more recent vintage 
in AI and it has been stimulated by AI work in “real life” 
situations. Work on this problem will be presented in the 
context of AI applications to scientific work in the papers 
by Stefik (molecular genetics) and Sridharan (psychology). 

Some of the systems presented in this session have al¬ 
ready reached high levels of expertise in limited domains. 


CASNET/glaucoma at Rutgers and INTERNIST at the 
University of Pittsburgh and PROSPECTOR at SRI are in 
this category. There are also several “younger” systems 
that are still in a process of intense development and growth. 
In some of them techniques developed/tried in previous 
knowledge-based systems are being assimilated and adapted 
(e.g., the structure of MYCIN 4 is being used in the PUFF 
project) and the DENDRAL 5 experience as well as ideas 
from several more recent AI efforts are being used in the 
MOLGEN project. On the other hand, in the BELIEVER 
task discussed by Sridharan, a new system, called AIMDS, 
is being developed to provide an appropriate environment 
for theoretical work on the task. 

We are now at a point where substantial achievements 
have been made in AI applications and there are several 
new efforts that are starting to build on top of past experi¬ 
ence. In general, there has been good progress in this area 
and the prospects for the future of high performance in AI 
expert systems are bright. 

The following is a summary of the papers to be presented 
at the NCC session. 
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THE DEVELOPMENT OF CLINICAL EXPERTISE IN 
THE COMPUTER—THE EVOLUTION OF 
CLINICAL DECISION-MAKING PROGRAMS AT 
MIT—Peter Szolovits 

We at the Clinical Decision Making Group at MIT’s Lab¬ 
oratory for Computer Science, along with our collaborators 
at the Tufts-New England Medical Center Hospital, have 
been engaged for several years in the development of com¬ 
puter programs which embody expert medical knowledge. 
We have applied techniques developed in the study of Ar¬ 
tificial Intelligence (AI) to encode expert clinicians’ ap¬ 
proaches to handling a number of important medical prob¬ 
lems: (1) prescribing appropriate doses of digitalis glycosides 
to patients with congestive heart failure or arrhythmias (the 
DIG program), (2) taking the history of the present illness 
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of a patient with kidney disease (the program is called PIP), 
and (3) diagnosing and treating patients with acid/base and 
electrolyte disturbances. We have also studied the applica¬ 
tion of decision analytic techniques to the testing and treat¬ 
ment of Hodgkin’s disease, the differential diagnosis of acute 
renal failure, the value of coronary surgery to the individual 
patient, and other problems. 

Throughout many of these projects, we have discovered 
that our initial estimate of the difficulty of the problem has 
been far from what we finally decided. Naturally, we have 
often underestimated the magnitude of the effort needed to 
collect data, generate plausible decision structures, and 
choose and build computer implementations for the data 
base and reasoning components of the program. Not so 
obviously, however, we have also found that some problems 
succumb to much simpler techniques than what we had 
originally estimated to be needed. 

The key to these observations, we now think, concerns 
the degree to which limited human-like expertise in a com¬ 
puter program is adequate to cover the great majority of 
situations which arise in the program’s domain. To the ex¬ 
tent that a simple statistical model or a small set of decision 
rules or protocols can be made to work, any program which 
uses those models will be straightforward. It is only when 
the task involves complex reasoning to deal with a wide 
variety of medical possibilities, when a “deep” understand¬ 
ing of a problem is needed, or when the medical situation 
can consist of a complex combination of many simultaneous 
difficulties that the programs we write (and the experts’ 
knowledge which with models) must be sophisticated. 

The presentation will review a number of attempts to build 
programs for digitalis therapy, to contrast them in terms of 
the magnitude of medical knowledge involved in each and 
the corresponding sophistication demanded of the computer 
implementation. 
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THE ROLE OF HYPOTHETICAL REASONING IN 
DIAGNOSTIC PROBLEM SOLVING—Harry Pople 

INTERNIST is a computer-based system designed to deal 
with complex diagnostic problems in internal medicine. An 
important aspect of this task domain is the need to synthe¬ 
size ad hoc diagnostic categories to characterize patients 


with two or more disease processes at one time. The pres¬ 
entation will review the role of hypothetical reasoning in the 
INTERNIST approach to this concept-formation component 
of the diagnostic task. 
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STRATEGIES OF GLAUCOMA TREATMENT 

PLANNING—Casimir A. Kulikowski and Sholom M. 

Weiss 

The ultimate goal of most difficult medical consultations 
is to produce advice that will help formulate a plan of treat¬ 
ment with the greatest likelihood of success for the patient. 
Recently, several computer consultation programs using 
Artificial Intelligence techniques have developed different 
strategies for treatment planning. 1-3 The strategies used in 
the CASNET/Glaucoma program. 2 ’ 4 are designed to select 
plans for the long-term management of complex glaucoma 
cases over many follow-up visits. 

Although many subjective rules of clinical judgment enter 
into the formulation of treatment plans, the expected out¬ 
comes are usually based on an understanding of the phys¬ 
iological effects of the treatment. In CASNET/Glaucoma, 
a causal model describes the mechanisms of disease as well 
as their expected changes following treatment. A two-level 
data structure is used: general treatment plans subsume or¬ 
dered sequences of specific treatments. The initial step is to 
select a general plan of treatment on the basis of the patient’s 
diagnostic categorization. The choice of specific treatment 
within the plan is determined by many of the individual 
characteristics of the patient, through the use of scoring 
functions for the enhancement and inhibition of selections. 
For example, the status of the fellow eye of a patient be¬ 
comes a crucial factor when surgery is contemplated: the 
potential risk may be too great if there is little or no vision 
in the fellow eye. 

For follow-up visits, the effectiveness of any current treat¬ 
ments must be judged and compared with the expected ben¬ 
efits of the alternatives suggested by the selection strategy. 
The diagnostic status of the patient must also be reassessed, 
in the case that changes dictate a complete change of treat¬ 
ment plan. In the event that the patient remains within the 
scope of the original general plan, but the original specific 
treatment is not sufficient to control the progression of dis¬ 
ease, the program selects the next, stronger medication in 
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the sequence. Potential side-effects or complications are 
monitored and may result in changing the order in the se¬ 
quence of specific therapies. 

The CASNET/Glaucoma program’s recommendations 
were compared to those of a panel of experts at the 1976 
meeting of the American Academy of Ophthalmology and 
Otolaryngology. 5 The resulting discussion is serving as the 
basis for testing a variety of new model-based strategies for 
treatment. 
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USE OF ARTIFICIAL INTELLIGENCE FOR 
INTERPRETATION OF PHYSIOLOGICAL 
MEASUREMENTS: PULMONARY FUNCTION 
DIAGNOSIS AND I.C.U. VENTILATOR 
MANAGEMENT—J. C. Kunz, L. M. Fagan, R. J. 
Fallat, D. H. McClung, J. S. Aikins, H. P. Nii, E. A. 
Feigenbaum and J. J. Osborn 

Interpretation of physiological measurement data is a cen¬ 
tral process in medical diagnosis. Biomedical engineering 
has succeeded in providing a wide variety of quantitative 
measures of physiological state, but the biomedical com¬ 
munity has made only limited effort at systematically at¬ 
tempting to automate the interpretation of the medical sig¬ 
nificance of these quantitative measurements. 

We characterize the problem of medical data interpreta¬ 
tion by its medical and its sociological aspects. We choose 
to highlight the following features of the medical problem: 

Judgmental: Deductive and inductive reasoning are both 
used to conclude a diagnosis given a medical situation; 
Uncertain: There is only a loose relation between meas¬ 
ured result, physiological state, overall clinical state, and 
effects of therapy; 

Redundant: Measurements have redundancy with each 
other; 

Context-dependent: Interpretation depends upon the clin¬ 
ical situation; 

Time-dependent: The value and meaning of information 
changes. 

In addition, we highlight the following features of the 
sociological situation in which medicine is practiced: 


Experience: Good clinical practice is characterized by 
experience, in addition to didactic knowledge; 
Interpretation: The consultant, either human or auto¬ 
mated, should explain the process followed in making a 
consulting diagnosis. 

Expectation: Patient condition is viewed as improving or 
deteriorating in relation to an expectation of disease 
course. 

We will describe a general architecure for a medical in¬ 
terpretation system which recognizes these features of the 
problem. We have chosen Artificial Intelligence (AI) tech¬ 
niques for our approach to medical data interpretation. AI 
offers an approach to symbolic reasoning which is amenable 
to automation. The symbolic manipulation allows us to rep¬ 
resent the knowledge of the clinical problem in a direct form, 
i.e., as structured relations among medical states and meas¬ 
urements. We report here on our early progress with use of 
AI to address two medical problems as we see them. 

The PUFF system is now in routine use in our hospital 
for interpreting pulmonary function test measurements. 
PUFF interprets the medical significance and implications 
of input quantitative test data and patient history. Using a 
production rule formalism, PUFF has a set of about 60 
physiological rules of the general form “IF condition THEN 
draw conclusion.” The rule formalism has allowed our pul¬ 
monary physiologist to define the medical interpretation 
problem in his terms, using a level of judgment and a com¬ 
pensation for the uncertainty and redundancy of the data 
which he feels appropriate. The system is implemented in 
a version of the MYCIN system on the PDP 10 and in a 
straightforward way on the PDP-11. The system operates in 
a batch mode, accepting input from the data collection com¬ 
puter and printing out interpretations. In a 144-case evalu¬ 
ation, the system showed 86-93 percent agreement with the 
physiologist in diagnosing the presence of major pulmonary 
disease. For comparison, the physiologist changed his initial 
diagnosis of the presence and severity of major pulmonary 
disease upon further consideration in 42 of 107 cases during 
an early validation process. 

The VM system is designed to interpret the clinical sig¬ 
nificance of quantitative measurements from our ICU pa¬ 
tient monitoring system. Every 2-10 minutes, VM will accept 
the 40 measurements comprising the monitoring data and 
make a context and time dependent interpretation of the sig¬ 
nificance of data. The system first compares measured data 
with expected values in order to identify the possible pres¬ 
ence of alarm conditions. The system has rules which define 
criteria for accepting data as physiologically indicative or 
possibly spurious. Next the system focuses its attention on 
the measurements which categorize the clinical state of the 
patient, and then it attempts to identify possibly advanta¬ 
geous therapies. VM now includes the capability to process 
rules describing optimal procedures for removing patients 
from mechanical respirators. The program will print sugges¬ 
tions to clinicians concerning optimal control of respirator 
settings and choice of specific respirator types. Detection of 
unexpected mechanical failures and potentially significant 
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changes in the patient’s physiological measurements will be 
interpreted and reported to the clinician. 


MACHINE INFERENCE FOR MOLECULAR 

GENETICS: METHODS AND APPLICATIONS 

—Mark Stefik and Peter Friedland 

An increasing amount of research in Artificial Intelligence 
is involved with difficult applications which require repre¬ 
senting and using significant amounts of specialized knowl¬ 
edge in a computer. Several researchers 2,4 have suggested 
that this represents a shift of paradigm in AI—from seeking 
performance via powerful general methods toward a recog¬ 
nition that many difficult processes are knowledge intensive. 
This view emphasizes the use of large amounts of specialized 
knowledge in a fashion that facilitates their effective use and 
interaction. 

The MOLGEN project at Stanford is an example of this 
trend in research. MOLGEN the joint effort of computer 
scientists and geneticists at Stanford, is broadly concerned 
with the application of symbolic reasoning to molecular ge¬ 
netics. Molecular genetics is a vigorous and rapidly devel¬ 
oping discipline with many logical and medium-sized prob¬ 
lems for potential computer application. The main thrust of 
the MOLGEN effort has been in the development of pro¬ 
grams to assist in the design of laboratory experiments. 
Experiment design requires information about problem solv¬ 
ing goals, available laboratory techniques, and strategies for 
combining these techniques to achieve the desired goals. 
Work in computer-assisted design has been concentrated in 
a limited class of analysis experiments and a limited class of 
synthesis experiments. 

Development of performance programs in the MOLGEN 
project has required work on several fronts. Experiment 
planning involves representation of many kinds of laboratory 
methods and objects. To avoid creating separate packages 
for each kind of knowledge, we have developed a schema- 
based representation package 5 with some fairly general ca¬ 
pabilities. This part of our work, which occupied much of 
our first year of research, was considerably influenced by 
recent work in representation languages. 1 At the same time, 
the process of planning experiments and resolving conflicts 
has drawn on recent work in problem solving. 6 Prior to 
beginning our work in this area, we did a substantial review 
of problem solving methods. 7 

Projects like MOLGEN foster the art of applying the 
principles and tools of AI research to bear on specific prob¬ 
lems. This usually involves adapting and combining these 
tools; it always involves comparing and selecting among 
alternative approaches. One of the motivations for these 
efforts is their utility as case studies in methodology. We 
believe that artificial intelligence will mature as a discipline 
only by confrontation of its basic methods with the chal¬ 
lenges of various applications. From the case studies are 
slowly emerging the bits of a theory, which consists of an 
increasingly well characterized collection of paradigms for 


solving problems and representing knowledge. One example 
of the results of a case study from our own work is in 
Reference 8. 
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COMPUTER-BASED CONSULTANT FOR MINERAL 

EXPLORATION—R. O. Duda, P. E. Hart, and 

R. Reboh 

For the past two years, we have been developing an in¬ 
teractive system of computer programs called PROSPEC¬ 
TOR that is intended to act as a consultant to field geologists 
on problems of mineral exploration. The performance of the 
system is based on an internal set of models of various types 
of mineral deposits, and on a taxonomy of rocks, minerals, 
and other geological concepts. The models and taxonomy, 
termed the knowledge base, are provided to the system over 
a period of time by a panel of expert geological consultants. 
The field geologist using the system furnishes a set of ob¬ 
servations, often incomplete and uncertain, about the par¬ 
ticular prospect under investigation. The principal tasks of 
PROSPECTOR are to compare the observations of the user 
against the models, to find the most promising matches, to 
identify for the user the most important missing observa¬ 
tional data, and finally to form an estimate of the likelihood 
that a deposit in fact occurs on the prospect. Viewed ab¬ 
stractly, PROSPECTOR attempts to aggregate thfb knowl¬ 
edge of expert geologists in order to apply that knowledge 
to particular mineral exploration problems. 

The models of mineral deposits are represented in a spe¬ 
cial data structure termed an inference network. A node in 
the network represents an arbitrary geological situation like 
“The prospect lies within 200 miles of a subduction zone” 
or “The ratio of pyrite to bomite on the prospect is less 
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than 1.” Each node has an associated probability. A subset 
of nodes are distinguished as model nodes; that is, a node 
in this subset represents the situation that a mineral deposit 
satisfying a model occurs on the prospect. 

Nodes in the network are connected by directed arcs. An 
arc corresponds to either a logical connective or to a prob¬ 
abilistic inference relating the nodes it joins. An important 
form of reasoning in PROSPECTOR is to propagate proba¬ 
bilities along arcs from one node to the next. The compu¬ 
tations for propagation are based on subjective probabilities 
furnished by the experts, and are described in Reference 1. 

The inference network formulation is particularly well- 
suited to our purpose because it allows models to be built 
up incrementally over time by defining new situations and 
relations. Each relation, together with the nodes it joins, is 
called a rule, and for this reason PROSPECTOR is termed 
a rule-based inference system. 

The situations described by the nodes in the inference 
network are frequently compounded out of simpler situa¬ 
tions; in the previous illustration the occurrence of pyrite 
and bomite are evidently the primitive geological situations 
of interest. It is often important to reason about these prim¬ 
itive constituents of a more complex situation. For ex¬ 
ample, the user may have already supplied the system with 
the observation that no sulfide minerals occur on the pros¬ 
pect. Since both pyrite and bornite are sulfides, PROSPEC¬ 
TOR should be able to infer that it is meaningless to inquire 
about their ratio. To support reasoning such as this, we 
represent each situation in the inference network in a more 
detailed structure called a semantic network. 2,3 The semantic 
network for the previous example would explicitly articulate 


a situation in which both pyrite and bornite are present and 
are in the prescribed ratio. Furthermore, additional arcs 
denoting the subset relation would link pyrite and bomite to 
the mineral taxonomy in general and to the sulfide minerals 
in particular. 

PROSPECTOR operates as a mixed-initiative system. At 
any time the user may take the initiative and supply the 
system with observational data or may request further in¬ 
formation of several sorts. When the user relinquishes ini¬ 
tiative, PROSPECTOR asks the user to establish by obser¬ 
vation the likelihood that a particular geological situation 
occurs on the prospect. PROSPECTOR selects situations— 
that is, nodes in the inference network—that will have the 
most pronounced effect on the probabilities of the model 
nodes. Thus, PROSPECTOR in effect pursues a strategy of 
trying to confirm or refute hypotheses about the occurrence 
of a mineral deposit on the prospect. 
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INTRODUCTION 

The design and implementation of automatic, general pur¬ 
pose problem solvers is a complex task. Although there are 
several possible methodologies in which to couch a theory 
of general purpose problem solving, a popular approach in 
recent AI research has been problem reduction. Problem 
reduction is a technique in which the problem solver’s goals 
are broken down into sets of (presumably) simpler subgoals 
where the problem solver’s task of discovering a solution 
for each is easier than discovering a solution for the main 
goal. 

Implicit in this assumption of simplicity of subgoal solu¬ 
tions is the requirement that each subgoal be treated inde¬ 
pendently. Otherwise, the advantage gained by specifying 
simpler subgoals is defeated by the requirement for contin¬ 
uous interaction among those subgoals’ solutions. Thus, an 
important aim of the research in general problem solving 
has been to discover schemes whereby autonomy of the 
subgoal solution process can be preserved. 

There are times, however, when interaction among 
subgoal solutions is inescapable, either because of the order 
in which the subgoals are specified or because of innate 
interdependence of subgoal solutions. A promising approach 
to handling subgoal interaction is to allow the subgoal pro¬ 
cesses to assume that they are independent of each other 
and to provide a meta-process which watches the progress 
of the subgoal solutions. The “watcher” can assure that the 
currently active subgoal process does nothing inimical to the 
accomplishments of other completed subgoal processes. 

There are several strategies that have been investigated 
to deal with the interaction of subgoals. I will concern myself 
in this discussion to the class of solutions that treats the 
subgoals sequentially, allowing each subgoal process to 
make use of results of solutions generated for previous 
subgoals. This approach is typified by the systems of Rieger 
and London, 1 Tate, 2 Waldinger, 3 and Warren. 4 Several other 
approaches to subgoal interaction in problem reduction 
exist. These include what could be called the parallel ap¬ 
proach (Sacerdoti 5 ) in which subgoals are solved independ¬ 
ently and then subgoal solutions are merged together, elim¬ 
inating redundant operations. A third approach could be 
called the hybrid approach (Dawson and Siklossy 6 ), a com¬ 


bination of sequential and parallel approaches, in which the 
plan is expanded by choosing that subgoal to work on next 
which least interferes with the already existing solution. 
Finally, there is the so-called linear solution of Sussman, 7 
which, like the parallel, assumes total subgoal independence 
at plan time, and corrects for interaction problems at exe¬ 
cution time. 

In the sequential type of solution, a powerful mechanism 
for maintaining a world model is required wherein each 
successive subgoal can determine the effects on the envi¬ 
ronment which have resulted from the sequence of actions 
thus far proposed by prior subgoal processes. Each subgoal 
process is autonomous in that it does not care what partic¬ 
ular solution was generated by previous subgoal processes. 
Nevertheless, it should be aware of the effects those solu¬ 
tions had on the environment. The modelling mechanism 
must be adequate to represent these alterations to the en¬ 
vironment. 

Within this setting, I will discuss a modelling technique 
called Dependency-Based (DB) modelling which relies on an 
explicit representation of justifications for beliefs held by 
the problem solver. Using these justifications, the DB mod¬ 
elling mechanism is able to determine the relevant lines of 
inference to pursue during problem solving. Then, I will 
discuss three particular problem solving difficulties and I 
will propose remedies which will revolve about the DB mod¬ 
elling technique. The three difficulties are: 

1. Subgoal violation detection—the ability to detect that 
a subgoal whose existence is still required by the main 
goal has been undone by the solution to a brother 
subgoal 

2. Descriptor binding—the ability to recognize the correct 
binding in the plan for certain objects which are known 
in advance only by descriptions of their features 

3. Maintaining a consistent world model—when subgoal 
violations are detected, the ability to relieve those in¬ 
teractions by modifying the sequence in which actions 
are applied. Demands are made on the modelling mech¬ 
anism to update efficiently the world model to reflect 
these changes. 

In following sections, I will motivate these problems and 
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discuss a proposed solution founded on a modelling mech¬ 
anism based on dependency nets. 


BACKGROUND 

Maintaining world models for problem solvers is a task 
which has received much attention. Much of this effort was 
stimulated by discussion of the “Frame Problem” by 
McCarthy and Hayes. 16 The Frame Problem points out the 
difficulty of representing the influence of actions on a prob¬ 
lem solving environment. 

The modelling mechanism I will describe is based on a 
representation called dependency nets. A dependency net is 
a graph whose nodes represent beliefs and whose arcs rep¬ 
resent logical justifications for those beliefs. Utilization of 
explicit dependency representations for world modelling is 
due to Fikes. 8 Dependencies have also been used to repre¬ 
sent problem solving assumptions to help control processes 
such as backtracking. 9 

The power of dependency nets as the basis for modelling 
mechanisms is derived from their ability to direct the infer¬ 
ence processes which draw conclusions from new beliefs 
about the world. Dependency nets provide a convenient 
structure for maintaining knowledge about derived relations 
among assertions. In this way, changes in belief can be 
efficiently accommodated into the model by allowing already 
derived relations to direct the propagation of information 
through the network. Duda, et al. 10 have investigated the 
propagation of information through networks of relations, 
although their primary focus was to deal with contributions 
of justifications with variable confidences to the overall 
strength of belief of a hypothesis. My model draws heavily 
on the notion of propagation of information through the 
network, though these capabilities have been enhanced to 
deal with problem solving difficulties they did not consider. 

A detailed investigation of the nature of dependency struc¬ 
tures has been performed by Doyle. 11 While his primary 
focus was on control issues (e.g., dependency-directed 
backtracking 9 ), mine is on modelling issues. Among the im¬ 
portant ideas he developed are the notions of well-founded 
support and non-monotonic deduction. Well-founded sup¬ 
port is justification of a belief which ultimately relies only 
on fully believed premises. This notion is exercised fully in 
the theory described in this paper. I have not dealt with his 
concept of non-monotonicity which states that the justifi¬ 
cation for belief of some assertions may be based on the 
non-justifiability of their negations or of certain other asser¬ 
tions. There are many fundamental differences between our 
systems, though, because of the differences in purpose (con¬ 
trol vs. modelling). 

Besides dependencies, there are several popular tech¬ 
niques for modelling the state of the world during problem 
solving. Among these are context-layered databases 14,17 and 
“plan structure” models. 3,5 They both have advantages in 
terms of efficiencies for specific operations which are de¬ 
scribed later. They are both deficient, though, when com¬ 
pared to a dependency-based model in that they do not have 


the ability to record already derived relations among asser¬ 
tions and to utilize those relations at appropriate times. 

THE DEPENDENCY-BASED MODELLING 

MECHANISM 

As described in the Introduction, there are difficulties 
inherent to the sequential problem reduction approach which 
arise when subgoals are not independent. Their remedies 
will be addressed within the DB modelling mechanism for 
the Commonsense Algorithm (CSA) problem solver. 1 

Dependency nets 

The DB modelling mechanism relies on a representation 
known as a dependency net. A dependency net is a network 
whose nodes are assertions representing beliefs held by the 
problem solver. The arcs of the network represent relations 
among the assertions believed by the system. In this way, 
not only are the system’s current beliefs represented, but so 
are the justifications for holding those beliefs. 

The dependency network representation provides the ca¬ 
pability to make explicit the interrelationships among facts 
and beliefs held by the problem solving system. The net¬ 
work’s power is derived from its ability to guide inference 
and deductive procedures when new facts are added to the 
network or, more importantly, when the truth value of one 
of the system’s beliefs changes. For example, assume the 
system believes a fact PI to be true. Suppose then, that an 
attempt to deduce the believed truth of P2 occurs (e.g., the 
problem solver wants to know if the goal P2 is already true). 
The deductive mechanism determines that P2 should be 
believed FALSE and furthermore, the reason is that a de¬ 
ductive rule was retrieved that stated if PI is true then P2 
is false.* These facts are explicitly represented in the net¬ 
work notation of Figure 1. The truth values required for the 
relationship between PI and P2 to be valid (the schematic 
truth values) are denoted on the arc. The believed truth 
values for PI and P2 are denoted inside the nodes (e.g., 
P1:T). 

A great deal of flexibility results from the explicit repre¬ 
sentation of dependency relations. For example, if some 
event occurs which causes the truth value of P2 to change 
to TRUE, the system knows it can immediately draw the 
inference that PI is false.** If support for PI is explicitly 
represented, then those supporting beliefs can be updated 
in a similar way. In this manner, an alteration in truth value 
propagates through the network from consequents to ante¬ 
cedents by a process I will call antecedent propagation. 

Let us consider a slightly more complicated example of 
antecedent propagation. Suppose, in determining the truth 
value of an assertion P, the deductive component returns 
that P should be considered FALSE because PI and P2 are 


* In the context of this discussion, deduction will be defined as the process 
of determining the believed truth value of a specified assertion. 

** I take inference to be that process which computes the consequences of 
the assertion of a belief or the change in truth value of a belief. 
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believed TRUE and P3 is believed FALSE. This relation is 
represented in Figure 2. In this case, if P should change 
truth value, it is known that at least one of PI, P2, P3 must 
also change truth value. Deduction could be applied to each 
to determine which had changed, and this truth value alter¬ 
ation could be propagated backward through the net as be¬ 
fore. In order that the model always be consistent, whenever 
the consequent of a dependency relation changes truth 
value, deduction must be applied to each of its antecedents. 

A great savings in computation could be achieved if the 
application of deduction to some of the antecedents of an 
altered consequent is avoided. A reasonable approach is to 
mark each of the antecedents PI, P2, and P3 as having 
uncertain truth value (Figure 3), and let this uncertainty 
propagate through the net. In this way, applications of de¬ 
duction can be postponed until they are deemed relevant. If 
uncertainty is ever propagated to an assertion which is con¬ 
sidered important, then deduction can be applied at that 
point and the effect of the result of this deduction can be 
propagated through the net. What makes an assertion im¬ 
portant in a problem solving environment is described later. 

Dependency net behavior when an assertion’s truth value 
is altered is not governed solely by antecedent propagation. 
Truth value alteration can be propagated from antecedents 
toward consequents (consequent propagation ), though this 
is logically different from antecedent propagation. For ex¬ 
ample, in Figure 2, if PI changes truth value then little can 
be said about the truth value of P. The support of belief P 



has been lost, but it may be that alternate support for the 
currently believed truth value of P can be derived. When 
support for an assertion’s believed truth value is lost, its 
truth value becomes uncertain. To determine if there was 
an alteration of truth value as a result of the lost support, 
deduction must be reapplied to an assertion whose truth 
value might not have changed at all. Therefore, all means of 
support for an assertion should be represented in the net. 

For any assertion P whose truth value is desired, the 
deductive component should not merely retrieve the first 
applicable rule discovered to provide support for P’s de¬ 
duced truth value, but all applicable rules. I will call the 
application of all valid rules, exhaustive support. Although 
exhaustive support places a heavy demand on the deductive 
component, it is crucial for some applications to which the 
dependency net will be applied during problem solving. Hav¬ 
ing exhaustive support applied to all assertions in the net¬ 
work guarantees the well-founded support which helps avoid 
the dangers of circular justifications (see Reference 11). 

Multiple applicable deductive rules occur frequently and 
are represented as in Figure 4. In this example, the falsity 
of P is independently supported by the truth of PA1 and 
PA2 and by the truth of PB1 and the falsity of PB2 and PB3. 
If some event should cause PB1 to be believed FALSE, the 
belief in the falsity of P does not change because of the still 
valid support provided by PA1 and PA2. 

If, as in Figure 2, a fact P has only singular support, then 
a change in truth value of an antecedent (say PI) signals a 
change in truth value of the consequent P. The actual sup¬ 
port for the new truth value of P must be determined by 
deduction, but one can hypothesize on logical grounds that 
PI has something to do with it. 

As with antecedent propagation, the modification of truth 
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values of beliefs in the dependency net can be propagated 
along dependency links, though in the direction of the con¬ 
sequent by consequent propagation. Rather than applying 
needless deduction, assertions whose truth values have be¬ 
come uncertain can be so marked with uncertain truth value 
until a solid determination of its truth value and new support 
relations is required.* 

A brief system overview 

The dependency-based model is intended to serve as a 
component of a complete problem solving system whose 
functional modules are depicted in Figure 5. The DB model 
can be considered as an intelligent interface lying between 
the planning and deductive components. The planning com¬ 
ponent frequently requires truth values for various asser¬ 
tions (e.g., goals) and those truth values are determined by 
the deductive component. The DB model serves as an in¬ 
termediary who remembers the results of deductions and 
who applies a highly constrained inference process for the 
purpose of maintaining a consistent set of beliefs about the 
problem solving environment. The only facts contained in 
the DB model are facts which are relevant to the problem 
solving environment because the model is activated only by 
assertions passed to it by the planning component. Addi¬ 
tionally, a desirable feature of this organization is that the 
DB modelling component can be studied independently of 
any particular planning module. 

Deduction rules 

Dependency relations represented in the network arise 
from two sources. The first of these sources is the set of 
assumptions made by the problem solver itself. Such as¬ 
sumptions include the expected consequences of performing 
actions, selection of objects to participate in plans, and 
action sequence relations (that the results of an action not 
only depend on the action but also the context in which the 
action is performed). 

The second source of dependency relations is the deduc¬ 
tive mechanism. Dependencies which arise as a result of the 
invocation of deduction represent logical relationships 
among beliefs. The schematic examples in the previous sec¬ 
tion are instances of logical dependencies. 

The representation of logical support rules in the depend¬ 
ency-based modelling scheme is rather straightforward. All 
deduction rules simply are represented as dependency net 
schemata. For instance, the dependency relation repre- 


Figure 5 


* Consequent propagation of uncertain truth values is similar to the truth 
maintenance technique utilized by Fikes. 8 
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sented in Figure 6 was generated from the deduction schema 
in Figure 7. 

For a logical relation between assertions to be valid, the 
assertions must be assigned the schematic truth values. The 
schematic truth values of the support rule in Figure 8 rep¬ 
resent that if P(x,y) and Q(y) are true then R(x) is false. The 
instantiation of this rule in Figure 9 illustrates an instance 
in which the relation is valid; the antecedents represent 
actual support for the consequent. Figure 10 shows an in¬ 
stance of this rule which still describes the same logical 
relationship among the predicates P, Q, and R (called po¬ 
tential support), though the rule is not currently valid since 
the truth values of the assertions do not match the schematic 
truth values required by the relationship. 

An example of dependency net behavior 

The goal of the application of logical support rules is to 
construct and maintain a richly interconnected network of 
assertions such that for any assertion P, every other asser¬ 
tion contained in the net which bears a logical relationship 
to P will be connected to P along some path in the network. 
These paths connecting logically dependent assertions will 
be discoverable by antecedent or consequent propagation. 
In order to maintain a consistent model, all valid alternative 
means of support as represented in the rules matching the 
derived truth value for a given assertion must be retrieved. 
For this retrieval, the particular method used by the deduc¬ 
tive search is not significant to the DB model. The intent of 
this modelling method is to limit the amount of deduction 
performed during inference while maintaining a consistent 
model. 

Example 

Let us now consider an example which describes how the 
dependency network behaves in order to maintain model 
consistency. Then, in the next section, we will see how this 
behavior can be “fine-tuned” to limit the number of deduc¬ 
tions to just those that are required. 

Consider the set of rules in Figure 11. Suppose that these 
rules are sufficient to maintain a consistent model in some 
domain. Suppose also that in this domain, the problem sol¬ 
ver possesses a strategy for achieving (making true) the goal 
predicate P3. This strategy is depicted in Figure 12. The 
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Figure 8 


Figure 9 


Figure 10 


strategy is encoded in the Commonsense Algorithm repre¬ 
sentation. 1,12 In this case the strategy pattern states that 
performing the action A causes the predicate P3 to become 
true (or the goal P3 to be achieved) provided at the time the 
action is performed that P5 is true. 

Now assume that in the initial world (the problem state¬ 
ment) Pi, P4, P5, and P10 are true (Figure 13). The problem 
solver attempts to determine if goal P3 is already true. If so, 
it need do nothing further. The truth value of P3 is deduced, 
a value of FALSE is returned, and the strategy of Figure 12 
is applied in an attempt to achieve it. 

The assertion P3 having truth value FALSE is entered 
into the existing dependency net and exhaustive support for 
this belief is determined and drawn into the net (Figure 14). 
In this case, RULE1 and RULE2 are retrieved since their 
consequents match P3 and they assert the falsity of P3. The 
general retrieval algorithm being applied assures that an 
instantiation of a support rule will be knitted into the net if 
the consequent and all antecedents which are already in the 
net match the schematic truth values required of them. 

Following the same line, the truth value of P2 should be 
deduced to determine whether the potential support pro¬ 
vided by RULE2 indeed represents actual support for P3: F. 



Suppose a truth value of TRUE is deduced for P2. The 
appropriate support rules for P2:T are retrieved and this 
process continues until the truth value of all assertions are 
known and exhaustive support is explicitly represented in 
the net. This dependency net, which shows P3 and all valid 
lines of support for belief in P3’s falsity, is displayed in 
Figure 15. 

The problem solver now applies itself recursively to P5, 
the subgoal specified by the strategy. Deduction of the truth 
value of P5 is hot required since it is already explicitly 
represented in the dependency net. Furthermore, no search 
for support is required since every assertion in the net is 
already exhaustively supported. Since the subgoal is solved, 
the action A can cause the truth of P3, our original goal. 
This is explicitly represented in Figure 16 by an action- 
consequent link. 

Inference behavior spontaneously occurs as the network 
reacts to this alteration in truth value of one of its assertions. 
Network inference amounts to an attempt to maintain con¬ 
sistency of the world model. By antecedent propagation, the 
inference process concludes that P4 must have become false 
(RULE3 from Figure 11 states this relationship between P3 
and P4 explicitly). Similarly, it is concluded that at least one 
of PI, P2 must have become false. The model update so far 
is depicted in Figure 17. Let us assume that when deduction 
is applied to PI and P2, it is concluded that only P2 has 
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become false. It follows, if P2 is false then so are P6 and 
P5. 

The propagation of alteration of truth values continues 
until the condition of the network stabilizes. The model is 
then consistent and the effects of performing the action A 
will have been accommodated. Every assertion in the net 
has an associated truth value which can at this point be 
considered to reflect the current state of the world as mod¬ 
ified by the action A. Exhaustive support is required in 
order to assure that the effects of an alteration of truth value 
are propagated to every assertion which bears any logical 
relationship to the assertions which are immediately repre¬ 
sented as being altered by performance of the action. In 
practice, the ability to maintain exhaustive support is limited 
by the capabilities of the deduction mechanism and the time 
allotted to search for lines of support. 


Application of rules 

If it were possible to be more selective about when to 
apply the deductive mechanism to determine whether a rule 
retrieved as potential support indeed represents actual sup¬ 
port, one could greatly reduce the overhead which would be 
required to maintain exhaustive support in large dependency 




Figure 15 


nets. In this section, I will describe modifications to the 
algorithm for verifying actual support which limit the amount 
of deduction required to maintain exhaustive support. The 
method for retrieving exhaustive support locates all support 
rules which provide potential support for the truth value 
determined by the deductive component for the consequent 
assertion. Those rules which do not provide actual support 
should not be weeded out until the need for actual support 
arises. This would occur, for instance, if the assertion 
changes truth value. 

Consider the situation in Figure 14. The truth value of P2 
is required to determine if RULE2 represents an actual 
support relation for P3: F. Recalling that the very purpose 
of exhaustive support is for drawing inferences from an 
alteration of truth value of an assertion in the net, we note 
that if P3 were never to change from false to true, there 
might never be a need to determine if RULE2 is indeed an 
actual support rule. The decision on the truth value of P2 
can therefore be delayed until P3 changes truth value. This 
will become a key source of efficiency in the DB modelling 
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scheme, the efficiency being derived from the ability to 
focus the deductive process on only those assertions cur¬ 
rently considered crucial to maintaining model consistency. 
In other words, potential support need only be verified when 
actual support is required. 

To illustrate this point, let us introduce three new rules to 
the set depicted in Figure 11. The new rule set appears in 
Figure 18. With this rule set, when RULE2 was applied to 
support P3:F, exhaustive support for P1:T would have de¬ 
manded the deduction of truth values for P8 and P9. This 
would have been a clearly wasteful operation since the truth 
value of PI was never altered. In addition, since P8 and P9 
were not already in the net the problem solver is not inter¬ 
ested in them anyway.* 

To realize this idea, the modifications to the information 
propagation algorithms are as follows. 


* It should be noted that intermediate assertions like P2 and P6 in Figure 16 
only serve the purpose of tying “important” assertions together. Important 
assertions are those stated in the initial world and those for which truth values 
are explicitly requested by the problem solver (i.e., goals). 


(1) Exhaustive Support—Knit into the dependency net all 
support rules whose consequent matches the assertion 
and its deduced truth value and whose antecedents 
which are already in the net match the schematic truth 
values of the rule. Mark any newly generated asser¬ 
tions (antecedents which were not already in the net) 
with **’, meaning that deduction has not been pursued 
on this node so that this rule provides actual support 
only if this node has proper truth value. 

(2) Antecedent Propagation—When an assertion A 
changes truth value: 

[1] Change truth value of nodes which are the sole 
antecedents of valid rules of which A is the con¬ 
sequent. 

[2] In rules which have multiple antecedents mark all 
the antecedents *?’ (meaning that the truth values 
of these assertions might have changed). If any 
nodes are now marked ‘?*’ then it is necessary to 
apply deduction to that node with respect to the 
net before the consequent was changed for the 
purpose of determining whether the rule was pro¬ 
viding actual support for the altered consequent. 
If the deduced truth value for the antecedent does 
not match that required by the rule, then the rule 
can be discarded because it did not represent ac¬ 
tual support. If the truth value of the antecedent 
matches the rule, then mark the antecedent as 
having uncertain truth value ‘?’ and put it on the 
antecedent propagation queue. 

These modifications minimize the number of deductions re¬ 
quired by inference in accommodating the alteration in belief 
of an assertion. This inference process determines all rele¬ 
vant effects of a modification in belief of an assertion by the 
problem solver. Deduction is only performed when required 
and the network representation is flexible enough to make 
bookkeeping for the process a fairly simple matter. 



Figure 18 
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A comment about deduction 

My purpose here has not been to propose a new mechan¬ 
ical theorem prover; I have intentionally ignored the prob¬ 
lem of deduction entirely. Rather, my purpose has been to 
demonstrate that the goal structure of a plan and a repre¬ 
sentation for the justifications of beliefs can be used to guide 
an efficient modelling technique. Deductions are performed 
only in response to requests for truth by the problem solver 
and in determining the actual support relations for believed 
assertions. One strength of this approach lies in its ability 
to avoid deduction for the maintenance of a consistent world 
model until the truth of assertions in that model have a 
bearing on the outcome of the planning process. 

There is an extensive and well-known body of literature 
on the subject of mechanical deduction. 13 The dependency- 
based modelling mechanism places no restrictions on the 
structure of the deductive process. The deductive compo¬ 
nent could be implemented as a mechanical theorem prover 
or as a search through the collection of support rules pro¬ 
vided. The DB model merely assumes the existence of some 
deductive capability which can determine the truth value of 
an assertion with respect to the context established by other 
known beliefs and that the determined truth value is com¬ 
patible with that which would be derived if the support rules 
were applied by some search process for the purpose of 
deduction. 


APPLICATION OF THE DEPENDENCY NET MODEL 

In the Introduction, I described three difficulties for gen¬ 
eral problem solvers that synthesize solutions to conjunctive 
subgoals by considering the subgoals sequentially. These 
difficulties arise when the subgoals are not independent. I 
will now outline solutions to these difficulties within the DB 
model framework. 


Subgoal violation detection 

If techniques are to be devised to deal with potential 
interaction among solutions to sets of conjunctive subgoals, 
then some method for detecting those interactions must be 
available. Remembering that one of our aims is to maintain 
as much autonomy as possible for the solution process of 
each subgoal, we require a scheme for subgoal violation 
detection which can run in the background without affecting 
the activity of the problem solver directly. Subgoals must 
remain intact until the main goal for which the subgoal has 
been defined has been achieved. A popular scheme for guar¬ 
anteeing the temporal scope of a subgoal has been to use 
protection. 1,3,7 

The violation detection scheme described here utilizes 
inference as a restricted search process for a general system 
of violation detection. This violation detection scheme de¬ 
rives naturally from the inferences which occur to accom¬ 
modate a change in belief in the net. To illustrate, assume 
a goal G has been solved by the problem solver. A protection 


violation would occur if, after G is asserted to be true (after 
it is solved), a consequence of that assertion is to make a 
protected assertion false. It follows that detecting protection 
violations can be accomplished by hypothesizing G to be 
true before actually making it so, allowing the dependency 
net to react to that assumption, then determining whether 
a change of truth value propagated to an assertion marked 
protected. The violation could be eliminated by processes 
such as described by Rieger and London, 1 Tate, 2 or Waldin- 
ger. 3 

It should be noted that this type of processing could con¬ 
ceivably be performed in a “context-based” modelling sys¬ 
tem. A new (hypothetical) context could be created, G as¬ 
sumed to be true, implications of this assumption 
determined, and any occurring violations discovered. If no 
violations were discovered, the hypothetical context could 
be popped and the implications about the assumption of G 
true would be discarded. 

The dependency scheme never throws away the work it 
has done in computing the implications of a given truth value 
for an assertion. When G actually becomes true as a result 
of some action proposed by the planning process, much of 
the effort expended by inference to determine the implica¬ 
tions of the assumption about G being true can still be used. 

In addition to its efficiency, I would argue that this is a 
more general violation detection scheme than has been pro¬ 
posed before (see References 2, 3 and 5, for instance). It is 
my feeling that utilizing a general inference process for vi¬ 
olation detection allows for application of the problem solver 
to more complex domains than would be allowed by a de¬ 
tection process which relies solely on syntactic pattern 
matching. Furthermore, the dependency structure helps 
focus the search for a violation. 


Descriptor binding 

It is often necessary during problem solving to defer de¬ 
cisions about the bindings of certain variables. These are 
variables for which bindings are not explicitly specified 
when a strategy pattern (Figure 19) is instantiated to satisfy 
a goal statement (Figure 20). Thus, the “CLEARTOP” 
strategy has a free variable which should be bound to the 
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GOAL: (CLEARTOP A) 



location at which the block on top of the block to be cleared 
should be placed. One advantage of being able to defer the 
choice of a binding for this variable is that it may not be 
obvious at the time the strategy pattern in Figure 19 is 
instantiated what the most appropriate choice for a binding 
would be. By deferring the decision, the problem solver can 
wait until enough information about a suitable binding has 
surfaced before making that decision. 

Consider the problem from Waldinger 3 depicted in Figure 
21. For the purpose of synthesizing a reasonably efficient 
plan, the problem solving system should have the ability to 
detect that in clearing A, the most suitable location to place 
B is ON C, thus solving the second subgoal in the process. 
Problems of this complexity have been handled by defining 
variables which can be manipulated by the problem solver 
as objects, and binding those variables to actual objects at 
the time at which a suitable determination of the correct 
binding can be made. 5,7 

The solution which I am proposing follows the same line 
of reasoning by binding variables which can be treated as 
objects. These variables are called descriptors because they 
can describe semantic restrictions on the eventual referrent. 
For example, consider the descriptor 

(*DESCRIPTOR -FREESPACE 

[OR (AND (CLASS -FREESPACE BLOCK) 
(CLEARTOP -FREESPACE)) 
(REGION-OF -FREESPACE TABLE)] 
[TABLE-REGION-1] ). 

This descriptor describes the object on which to place a 
block which has been picked up for the purpose of clearing 
another. It is a variable with semantic restrictions on its 
eventual binding specifying that FREESPACE is either a 
block whose top is clear or it is some region of table. If no 



(AMD (ON A B) (ON B C)) 

- > 



binding can be discovered during plan synthesis, the table 
is declared as the default. 

The relationship of descriptor binding to the dependency 
network is as follows. All assertions which refer to a de¬ 
scriptor are included in the dependency net. In the example 
above, the assertion (ON B -FREESPACE): T would be 
included in the net as a result of clearing A when achieving 
(ON A B). If the problem solver makes a request for the 
truth of a goal (e.g., (ON B C)) which matches an assertion 
in the dependency net containing reference to a descriptor 
(e.g., (ON B -FREESPACE)), then a candidate for a de¬ 
scriptor binding has been found. The potential binding object 
(C) must still pass a few tests. If the object which matched 
successfully against the descriptor also satisfies the semantic 
restrictions specified in the descriptor, and it satisfies all 
other assertions in the net which reference the descriptor, 
then the object can be bound to the descriptor. This binding 
technique is similar to Sacerdoti’s “Use Existing Object” 
critic. 5 A scheme is currently being considered to allow 
binding to occur even though not all the restrictions are met, 
and to allow the satisfying of those restrictions to be solved 
as subgoals by the problem solver. 

Descriptor binding amounts to an assumption made by the 
problem solver about an appropriate object to participate in 
the plan. The choice must be carefully determined, and will 
depend on facts known to the system. As such, the decision 
can be knitted into the dependency net as a new type of 
dependency. The advantage of making descriptor binding 
explicit within the dependency framework is that it allows 
the descriptor binding mechanism the same flexibility as 
other inference processes in the system. The choice of de¬ 
scriptor binding is considered best at the time the choice is 
made. Subsequent problem solving might prove that choice 
inferior to another. Since the binding is just a type of de¬ 
pendency, these decisions can be “unravelled” as easily as 
action sequence decisions to be described in the next sec¬ 
tion. Furthermore, the network treats descriptors as objects 
and can perform inference based on the features specified 
in the descriptor as though it were a real object. 


Modifying the action sequence 

A common operation in problem reduction problem sol¬ 
vers is to modify the partial action sequence as a plan is 
being synthesized. This modification is most often utilized 
by “sequential” type systems (those which deal with the 
subgoals sequentially) and is applied to relieve certain types 
of subgoal interactions. The particular types of interaction 
which can be relieved in this way are those which result 
from an incorrectly ordered action sequence (a possible side- 
effect of the assumption of subgoal independence). 

When subgoals are attacked independently, it is difficult 
to anticipate the proper order in which they should be 
solved. Often, in fact, an inherent interdependence among 
subgoals demands that the solutions that are independently 
derived for each be somehow melded together. A classic 
example of subgoal interdependence is depicted in Figure 
22. Regardless of the order in which the subgoals (ON A B) 
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GOAL: 


(AND (ON A B) (ON B C)) 

C 
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Figure 22 


and (ON B C) are solved, a reasonable plan cannot be 
synthesized by considering the two subgoals independently. 
A reasonable approach is to allow the subgoals to proceed 
sequentially and if a subgoal is violated by a subsequent 
subgoal to modify the action sequence to relieve the viola¬ 
tion.* 

Operations such as inserting a new action in the middle 
of an existing action sequence or removing an action from 
the middle of an action sequence must be acceptable to the 
modelling mechanism. This is a task which is difficult for 
chronological “context-based” models to deal with. The 
purpose of the modelling mechanism is to reflect the antic¬ 
ipated state of the world as it would be when modified by 
the sequence of actions being proposed by the problem 
solver. Let us examine an action sequence modification 
which involves removing an action, Ai, from an action se¬ 
quence {Al, . . . , An}, where n>i. The action-consequent 
links for the actions Ai, . . . , An can be removed and the 
inference mechanism can update the network to reflect these 
changes. The effects of actions Ai+1 through An can then 
again be accommodated efficiently, making use of the al¬ 
ready existing network structure created when they were 
originally introduced. 

Frequently the effects of actions are very local in the 
sense that other actions have little effect on the assertions 
effected by a particular action.** The power of the DB 
model for the operation of action sequence modification is 
derived from the fact that, often, many of the effects of the 
actions subsequent to Ai are little changed by the removal 
of Ai from the sequence or the insertion of a new action into 
the sequence. Since the effects of actions are explicitly rep¬ 
resented in the dependency net, much of the effort neces¬ 
sarily expended by a chronological model in recovering to 
a state of consistency after removal (or addition) of an action 
from the action sequence can be avoided. 

Let us examine some of the problems of using a chrono¬ 
logical “context-based” model for these operations. Figure 
23 shows a sequence of context frames in a context-layered 
database (a la CONNIVER 14 ) where each frame represents 
the assertions required to accommodate subsequent actions 

in the proposed action sequence {Al. An}. An efficient 

operation for the context-based models is to revert to a prior 
state, say the state just before inserting Ai on the action 
sequence (Figure 24). This operation would be necessary if 
one desired to remove Ai from the action sequence (and its 


* There are several schemes for determining an appropriate action sequence 
modification to relieve subgoal protection violations. See References 1-4. 

** See Stallman and Sussman 8 for an analogous discussion of the influence 
of particular electronic components on the overall behavior of a circuit. 


effects from the model). However, if one now tries to include 
the subsequent actions in the model (excluding Ai), all the 
work already performed in accommodating A/+1 through 
An must be reproduced (Figure 25). This is because context¬ 
layered databases allow a tree-like structure of context 
frames (as in Figure 25) while what is ideally required is a 
graph structure (Figure 26).* 

Modelling mechanisms other than “context-based” exist 
which handle the action sequence modification problem. 
Most notable are the “plan-structure” schemes of Sacerdoti 5 
and Waldinger. 3 Both these schemes make use of the exist¬ 
ing plan for computing the model. Both offer no more flex¬ 
ibility than dependency nets for recovering to prior states 
but they suffer, as do the context models, from being “for¬ 
getful” about inferences which have been made since the 
point at which the action sequence is being modified. In 
Waldinger’s model, this forgetting is deliberate because he 
does not want the problem solver to be committed to belief 
in an assertion derived on the basis of the current structure 
of the plan. The lack of commitment to derived assertions 
is required since the plan structure on which the derivation 
was based may be altered by a modification to the action 
sequence. Dependency nets allow the possible alteration of 
belief in an assertion due to modification of the plan to be 
explicitly detectable. Thus, beliefs whose derivations are 
based on the current structure of the plan can be incorpo¬ 
rated into the DB model (whereas in a plan-structure model 
they cannot) because the dependency net provides a means 
to detect which beliefs are affected by a modification to the 
plan. 

FUTURE WORK AND IMPLEMENTATION STATUS 

The ideas presented here are currently under implemen¬ 
tation. Among the immediate problems is that of restricting 
the bindings of variables for instantiations of selected sup¬ 
port rules. It is often possible for the consequent of a rule 
to match a newly entered assertion, while its antecedent 
matches no assertion in the net. This can leave unbound 
variables in assertions for which truth values are believed. 
For the rule in Figure 27, the consequent (CLEAR B):F 
matches (CLEAR -Y) and so (ON—XB):* is asserted. I 
plan to use the descriptor mechanism to specify the range 
of values the universally quantified variable—X might be 
bound to. Further, I intend to investigate the possibility of 


* It should be emphasized that context-based chronological models are more 
efficient for certain operations than the DB model. These operations are (1) 
the recovery to prior states (discussed above) and (2) the ability to transfer 
from one line of reasoning to another. For example, if it is later discovered 
that it would have been better to leave Ai in the action sequence, a context- 
based model provides for efficiently returning to that line of reasoning (i.e., 
transferring from point X in Figure 25 to point Y). These operations cannot be 
performed as efficiently by dependency net models. A possible compromise 
(which has not been pursued) would be to implement the dependency net in 
a context layered database for the specific purpose of making these desired 
operations efficient within the dependency net domain. It would be, however, 
a space-wise inefficient implementation, compounding the space-wise ineffi¬ 
ciencies of both dependency nets and context layered databases (e.g., CON¬ 
NIVER). 
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invoking the non-monotonicity assumption of Doyle 11 in 
order to limit the requirements for redeductions of truth 
values for assertions which have lost all support. 

The implementation strategy I have adopted is to associate 
with each arc in the net a set of associatively triggered 
computation units (i.e. demons), 15 whose purposes are to 
propagate information across the arcs when appropriate. 
Making the dependency net links be active components is 
an idea borrowed from the CSA mechanisms simulator. 18 In 
my case, this technique has the advantage of allowing the 
implementation of network capabilities to proceed in a very 
modular fashion. An alternative would be to provide a pro¬ 
gram which interprets a declarative representation of the 
network, but would require the integration of all the network 
capabilities into a single program. 

CONCLUSIONS 

This discussion has centered on dependency nets as a unified 
formalism for modelling in general problem solving systems. 


(ON - 

X -y) 

t 


f 

\ 

* 

(CLEAR -y) 


Figure 27 


I have suggested how such a modelling system behaves and 
how some enhancements to the basic ideas allow them to 
operate efficiently. Dependency nets serve to limit the 
amount of deduction required while performing inference 
for determining the important consequences of acquiring or 
modifying beliefs. 

Various competing solutions have been presented for 
some of the problem solving difficulties discussed. Most 
were presented within the context of complete systems with 
well defined capabilities. I feel that enough has been learned 
about the problems of general problem solving that some 
investigations can be initiated to research specific tech¬ 
niques independent of any particular complete problem solv¬ 
ing system. This endeavor demands some formalism in 
which these techniques can be described. I intend to pursue 
the idea that for many problem solving problems, depend¬ 
ency nets are an important formalism. 
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Error recovery in robots through 
failure reason analysis* 
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INTRODUCTION 

In performing a task, a robot may encounter a variety of 
unforeseen circumstances which prevent successful comple¬ 
tion of the task. Even simple actions, such as reaching out 
to grasp a rock, cannot be guaranteed to succeed without 
dynamic sensory feedback such as visual monitoring. For 
example, the hand may bump into the rock because of ma¬ 
nipulation errors or because of erroneous information about 
the location of the rock. Failures such as these occur be¬ 
cause sensors and effectors are inaccurate, and because the 
world in which the robot operates cannot be characterized 
exactly. 

Our goal is to design a robot that can recover gracefully 
and intelligently from failures that occur during the execu¬ 
tion of tasks in a static world with no other agents of change. 
The technique for recovery planning discussed in this 
paper is based on knowledge about failures—why they 
occur, what can be done about them, etc. This knowledge 
is used to determine the cause of a failure and to devise a 
recovery plan. Similarities and differences between this and 
other approaches are discussed in a subsequent section. 

A computer program called MEND has been implemented 
for exploring the above concepts. Though MEND only au¬ 
tomates recovery from failures in simple manipulation tasks 
that are performed by the JPL robot, 8,9,17-19 the techniques 
used in MEND are more widely applicable. 

A DISCUSSION OF THE APPROACH 

Execution of a task can be viewed as a sequence of trans¬ 
formation that changes the world from an initial state, S 0 , 
into a final state, S n , which represents the goal of the robot. 

Figure 1 depicts the intermediate states of the transfor¬ 
mations from S 0 to S n that result from the execution of a 
plan of actions, A 0 . . . A n _!. Assume that a failure is de¬ 
tected on execution of action A,. This means that the robot 
recognizes that action A* has not produced the expected 
state S i+1 , but rather has resulted in some failure state S f . 

The problem of error recovery is that of going from the 


* Now employed at Bell Laboratories, Naperville, Illinois. 


failure state S f to the goal state S n . One approach is to treat 
S f as though it were some arbitrary state and respond to the 
failure by replanning to achieve the given goal. In such an 
approach, there is no essential difference between error 
recovery [the transformation from S f to S n ] and planning 
[for instance, the transformations from S 0 to 5„]. Parts of 
the old plan can sometimes be reused, as for instance, by 
replanning from 5/ to Sj so that the original plan can be 
used to go from S, to S n . 

This paper suggests that by finding an explanation of the 
failure, i.e., understanding why action A* resulted in the 
state S f , a robot can determine where the problem lies and 
what can be done about it. With this viewpoint, error re¬ 
covery is seen as something quite different from traditional 
planning in that we do not merely ask, “What can be done 
to go up from S f to S n ?,” but begin by asking, “Why did 
action A* result in the state S f V’ in order to recover from 
the failure. 


OVERVIEW OF MEND 

Let us consider an example to illustrate the problems 
involved in recovery from failures, and some of the features 
of the solution incorporated in MEND. The robot hardware 
that is controlled by MEND consists of two TV cameras, a 
laser rangefinder, a manipulator, and a vehicle (Figure 2). 

Suppose that the robot is given the simple task of picking 
up a rock within immediate reach. Performing this task in¬ 
volves locating the rock using the TV cameras, opening the 
hand, positioning the hand around the rock, and grasping 
the rock (Figure 3). Assume that the hand bumps into the 
rock when attempting to position the hand around the rock. 
In this example, the manipulator system triggers the failure, 
bringing MEND into action. In other cases, MEND has to 
make explicit checks of certain conditions to detect failures. 
MEND’s execution monitoring is pragmatic in the sense that 
only those conditions that are easy to check on the basis of 
the information available directly from feedback, sensors, 
etc., are tested. A consequence of this lack of comprehen¬ 
siveness is that failures can propagate and may be detected 
only at a subsequent step. 

Having detected a failure, MEND is confronted with the 
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problem of dealing with the unexpected event. A heuristic 
used in MEND suggests that recovery actions can be found 
by determining why the failure occurred. The process of 
finding an explanation for the failure is termed failure reason 
analysis. Knowledge necessary for failure reason analysis 
is provided through a failure reason model associated with 
each action. The failure reason model represents knowledge 
about different reasons for failure of an action, how they 
may be distinguished, and what can be done to recover from 
a specific kind of failure. 

MEND represents possible explanations for a failure in a 
structure called the failure tree, an example of which is 
shown in Figure 4. The tree consists of a linked set of failure 
nodes and action nodes. There are four basic types of failure 
reasons that are dealt with in MEND; namely operational 
errors, information errors, precondition errors, and con¬ 
straint errors. Each failure node is associated with a failure 
type and a specific reason for failure. Action nodes are 
linked to an action in the plan being executed. An example 
of an explanation represented in Figure 4 is that the goal 
WITHIN-GRASP (node G) was never achieved by action 
MOVE-HAND-TO-GRASP, because of an operational 
SERVO-ERROR (node FI). 

Analysis of failures can now be thought of as the process 
of limiting the set of all possible explanations to the specific 
one that applies in a particular situation. MEND does this 
by limiting the set of failure reasons to be considered at each 
action node in the failure tree. Preconditions and constraints 
that have been verified on the basis of feedback information 
and sensor outputs, can be eliminated from consideration as 




Figure 3 


possible reasons for failure. More importantly, the distinc¬ 
tive features of the failure are used to limit the set of failure 
reasons. Figure 5 shows, for instance, a simple test and the 
implications of its results in eliminating some failure reasons 
from consideration. 

Once an explanation has been found, recommendations of 
corrective steps can lead MEND to successfully plan recov¬ 
ery from failures. For instance, if the failure is attributed to 
an information error (Figure 4) recovery can be achieved by 


OpormUonol Error 
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FAILURE TRIGGERED 
BY BUMP INTO ROCK 



positioning the hand near the rock, determining the location 
of the rock relative to the hand, and then repositioning the 
hand around the object (Figure 6). 

Other scenarios illustrating MEND’s capabilities are de¬ 
scribed in a thesis 14 that is a more complete documentation 
of the research study discussed here. The following sections 
describe the conceptual basis behind MEND and its design. 

A CLASSIFICATION OF FAILURE REASONS 

MEND’s analysis of failures is based on a classification 
of failure reasons into four categories. These are operational 
errors, information errors, precondition errors, and con¬ 
straint errors. Each is discussed in turn. 

Actions can fail to achieve their intended result because 
of certain inherent problems in executing the action. These 
errors are peculiar to the specific operation being performed 
and are, therefore, referred to as operational errors. For 
instance, the manipulator may deviate from the planned 
trajectory in moving from one location to another. This 
servoing error will not often have any serious consequences, 



but in some cases it can cause the manipulator to bump into 
the object to be grasped. 

An inaccuracy in the location of an object is an example 
of an information error. Though information errors can usu¬ 
ally be traced back to operational errors in the process 
being used to provide the information, they are to be distin¬ 
guished from such operational errors. As in our previous 
example of operational errors, information errors will not 
always cause failures and the robot may successfully com¬ 
plete its task despite such errors. 

Before executing an action, MEND will check precondi¬ 
tions and constraints. If any of these are found to be false, 
it will attempt to make them true before continuing. If all 
the preconditions of actions could be checked and verified, 
there would be no reason for failures to occur because of 
precondition errors. 

However, it would be unrealistic to expect a robot to 
check preconditions exhaustively, since some of them are 
inherently difficult to check. Therefore, the execution mon¬ 
itoring in MEND does not require that all preconditions be 
absolutely verified before executing an action. In some 
cases, executing subsequent steps may be the simplest way 
of finding out that they were or were not satisfied. For 
instance, checking to see that an object is truly within grasp 
before actually grasping the object is unnecessary and ex¬ 
pensive, since the very action of grasping will immediately 
indicate whether this is true or not. Even when previous 
actions were executed with the intention of making the pre¬ 
conditions true, the robot cannot depend on them being true 
since these actions themselves may fail. In any case, the 
consequence of not verifying preconditions is that after ex¬ 
ecution of an action, failure can be attributed to the falsity 
of preconditions. In this report, the term precondition errors 
refers to the above interpretation, to be distinguished from 
precondition failures detected before execution of an action. 

Finally, an action can fail because there are some con¬ 
straints that must be satisfied for successful execution. An 
example of this is that the object should be movable before 
the robot can transport it from one place to another. In a 
sense, these are merely special cases of preconditions that 
the robot has no way of making true. 


THE FAILURE TREE 

The structure of a failure tree, as illustrated in Figure 7, 
is described next. There are two types of nodes in a failure 
tree, failure nodes and action nodes. Failure nodes are rep¬ 
resented by circles and action nodes by rectangles. Action 
nodes are linked to a node in the execution trace through a 
plan link that is not shown in the preceding figure. Each 
failure node identifies the failure and its type, and may point 
to one or more action nodes through one of several kinds of 
links. The NAB (Never Achieved By) and IPB (Incorrectly 
Provided By) are two examples of such links. Action nodes 
point to failure nodes through a PRFF, the “Possible Reason 
For Failure” link. 


Figure 6 
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The example in Figure 7 can now be interpreted in the 
following manner: Goal G was never achieved by action A 
because of one (or more) of two possible reasons, FI and 
F2. FI indicates that the failure could have been the result 
of operational error, while F2 represents data incorrectly 
provided by action B. 

The types of failure associated with failure nodes are the 
ones discussed earlier (operational, information, precondi¬ 
tion, and constraint errors) with one addition. There is a 
special failure type which occurs only at the root of the tree 
and this represents a goal failure. 

Operational and constraint failure nodes do not point to 
other nodes since they are local to the failure action. Pre¬ 
condition nodes may be linked to action nodes that were 
intended to achieve the precondition. When such links are 
missing, it means that no action was executed to achieve 
this condition. It, therefore, represents an initial condition 
error. Similarly, information failure nodes are linked to ac¬ 
tions that provide the needed information, and missing links 
indicate initial information errors. 

BUILDING A FAILURE TREE 

The idea behind failure reason analysis is to find an ex¬ 
planation of the failure. By explanation we mean a chain of 
reasons represented by a path from the root node of the 
failure tree to one of its leaf nodes (G - A - F2 - B - F3 in 
Figure 7). If we consider all possible reasons for failure at 
each action node, then the failure tree represents all possible 
explanations for the failure. Clearly, only one or a few will 
be relevant in any particular situation. 

We next investigate what can be done to constrain the set 
of explanations. An execution trace in which unverified pre¬ 
conditions and constraints are noted provides the informa¬ 
tion necessary for determining whether failure can be attrib¬ 
uted to the falsity of these conditions. Note that a 


precondition is not considered to be verified merely because 
a previous action was executed to make this condition true. 
Instead, it is considered to be true when independent tests 
verify the truth of the precondition. To consider a trivial 
example, OPEN-HAND is verified by checking that the joint 
feedback information indicates that the fingers are fully sep¬ 
arated. Typically, other robot systems verify preconditions 
by looking for explicit assertions in the data base or by 
deducing the truth of these conditions from other assertions. 
The result of this is that no distinction can be made between 
those preconditions that can be verified directly from feed¬ 
back information and those that are only surmised to be true 
because of the intended effects of previous actions. 

A second important way in which the failure nodes 
branching from an action can be cut down is by looking at 
the manifestation of the failure. By identifying distinctive 
features of the failure situation, some of the reasons for 
failure can be shown to be impossible or at least unlikely. 

Third, the history of actions can show that certain expla¬ 
nations are impossible. If, for instance, no reasons for the 
failure of action C in Figure 8 can be found, we can conclude 
that B could not have failed because of F2. If there are no 
other reasons for the failure of B besides F2, we can con¬ 
clude further that A could not have failed because of FI. 
Such reasoning can significantly cut down the set of possible 
explanations, thereby pointing out the real explanation of 
the failure. 

Failure reason analysis in MEND, i.e., the process by 
which MEND determines the plausible explanations of the 
failure by building a failure tree, is outlined below: 

1. Find the set of possible failure reasons. 

2. Eliminate verified preconditions and constraint failures 
from this set by looking at the execution trace. 



Figure 8 
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3. Look for distinctive features of the manifestation to 
further constrain the set of failure reasons. 

4. For each of the remaining failure reasons, create failure 
nodes and link the action node to these newly created 
failure nodes. 

5. For precondition failure nodes, find the previous action 
responsible for making the precondition true from the 
execution trace. For information failure nodes, find the 
previous action that provides the necessary informa¬ 
tion. Create an action node for each of the previous 
actions so determined. 

6. Repeat the above process for the newly generated ac¬ 
tion nodes. 

7. Clean up the failure tree by eliminating impossible ex¬ 
planations (i.e., apply the process described in con¬ 
nection with Figure 8). 

THE FAILURE REASON MODEL 

The failure reason model is a model of actions which 
provides the knowledge necessary to perform failure reason 
analysis by means of the algorithm outlined in the previous 
section. First, the model should identify the set of possible 
failures. Operational failures are represented explicitly 
through a set of associations as shown below: 

(OPERATIONAL-ERROR MOVE-HAND-TO- 

GRASP(ROCK) SERVO-ERROR) 
(OPERATIONAL-ERROR MOVE-HAND-TO- 

GRASP(ROCK) COLLISION) 
(OPERATIONAL-ERROR LOCATE(ROCK) INACCU¬ 
RACY) 

Knowledge about other types of failures is implicit in the 
model of actions used in MEND, which specifies precon¬ 
ditions, constraints and needed information. For instance, 
any unsatisfied precondition is recognized as a possible rea¬ 
son for failure by MEND. Figure 9 shows the correspond¬ 
ence between the action model and the failure tree. 

The knowledge necessary to eliminate some possibilities 
on the basis of their manifestation is highly domain depend¬ 
ent. In MEND this knowledge is incorporated in a routine 
called the FRA-ROUTINE which is associated with each 
action. The FRA-ROUTINEs test for the presence or ab¬ 
sence of distinctive features of the failure in the current state 
of the world. These tests indicate that certain failure reasons 
are impossible, that others are likely, and so on. 

Figure 10 represents the FRA-ROUTINE associated with 
the action MOVE-HAND-TO-GRASP. At each node in the 
tree, a condition is tested and the appropriate branch taken. 
The leaf nodes of the tree specify a list of failure reasons 
that are considered possible in a specific context. 

We note several important points about these routines. 
As with precondition testing, the conditions being tested are 
easily computed using the data in the world model and the 
feedback information. For instance, the condition NEAR¬ 
DESTINATION is checked by verifying that the final po¬ 
sition of the hand is within a prespecified limit of the planned 
destination. 


FI : (OPERATIONAL-ERROR MOVE-HAND-TO-GRASP(ROCK) SERVO-ERROR) 
F2 : (OPERATIONAL-ERROR MOVE-HAND-TO-GRASP(ROCK) COLLISION) 

F3 : (NEEOED MOVE-HAND-TO-GRASP(ROCK) LOCATION(ROCK)) 

F4 : (CONSTRAINT MOVE-HANO-TO-GRASP(ROCK) GRASPABLE(ROCK)) 

F5 : (PRECONDITION MOVE-HAND-TO-GRASP(ROCK) EMPTY-HAND) 

F6 : (PRECONDITION MOVE-HAND-TO-GRASP(ROCK) OPEN-HAND) 



Figure 9 


The routine illustrated in Figure 10 assumes that the pre¬ 
conditions EMPTY-HAND and OPEN-HAND have been 
made true, and does not consider how a failure resulting 
from the falsity of these conditions will manifest itself. This 
simplification is possible because MEND’s execution mon¬ 
itoring will detect these precondition failures and will not 
allow execution of this action to continue unless the pre¬ 
conditions have been satisfied. Note that the FRA-Routines 
represent a procedural incorporation of knowledge about 
the manifestations of failure. 

There is a problem in the use of such routines to limit the 
set of failure reasons in Step 3 of the algorithm presented in 
the previous section. This routine requires information about 
the state of the world in making its decisions, which means 
that previous states of the world need to be represented. To 
keep this information around would be quite unrealistic in 
any large system, even using a stack-like mechanism for 
representing only the incremental changes. 

There are two ways of tackling this problem. Execution 
monitoring can be extended by running the FRA-ROU- 



. SERVO-ERROR • LOCATION-ERROR SAT - JOINT SATURATION 

• LOCATION-ERROR • NOT-QRASPABLE ND — NEAR DESTINATION 

. NOT-GRASPABLE PESV - POSITION ERROR ALONG 
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Figure 10 
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TINEs regardless of whether or not a failure occurred and 
marking the failure possibilities in the execution trace. 
Later, if failure reason analysis is necessitated in a subse¬ 
quent action, this possibility list presents a summary useful 
for analysis. 

A different strategy is to ignore the possibility of failure 
when there is no explicit triggering of the failure, and to 
later determine the failures that escaped detection. This is 
achieved in MEND by the representation of facts such as 
the one shown, 

(PTRIG (NEEDED MOVE-HAND-TO-GRASP(ROCK) 

LOCATION(ROCK)) NTRIG) 
where PTRIG=POSSIBLE-TRIGGER 
and NTRIG=NULL-TRIGGER, 

which says that a location error in the position of the rock 
could go untriggered when executing MOVE-HAND-TO- 
GRASP. (The reason for this is that if the location error is 
sufficiently large, the hand will completely miss the object.) 
With these facts in hand, MEND can immediately determine 
the set of failure reasons to be considered in Step 3 of the 
previously outlined algorithm, without having to run the 
FRA-ROUTINE for previously executed actions. 

An assumption implicit in the latter approach is that any 
triggered failure has already been handled. The first ap¬ 
proach is a more general technique but will perform unnec¬ 
essary computations in situations where plans are success¬ 
fully executed. 

There is one other aspect of the failure reason model. It 
provides information about how to correct a particular kind 
of failure. 

Knowledge necessary to correct operational errors is local 
to an action, and is represented in MEND by facts, such as, 

(TO-CORRECT (OP-ERROR MHTG(ROCK) SERVO- 

ERROR) MHTG-REC-PLAN l(ROCK)) 
where OP-ERROR=OPERATIONAL ERROR 
MHTG=MO VE-HAND-TO-GRASP 
and MHTG-REC-PLAN 1=(ROCK) 

BEGIN 

MOVE-HAND-TO- 

GRASP(ROCK) 

END, 

which says that an operational error of servoing in MO VE- 
HAND-TO-GRASP can be corrected by executing the plan 
MHTG-REC-PLAN 1. This plan will attempt recovery by 
simply repositioning the hand, and this strategy is appropri¬ 
ate for servoing errors. In general, TO-CORRECT steps are 
provided for each operational error. 

Constraint errors on the other hand, have no such “im¬ 
perative knowledge” 6 about what to do, since the robot does 
not have the capability of correcting such failures. If failure 
is attributed to a constraint error, the plan is aborted and 
MEND seeks human aid. 

Precondition and information errors are not directly as¬ 
sociated with corrective steps and are treated somewhat 
differently than operational errors. For precondition errors. 


the obvious thing is to reachieve the failed precondition 
before trying the action again. But before this can be done, 
some of the effects of the failed action may have to be 
undone. After the precondition has been achieved, other 
steps may need be redone to get execution back on the right 
track. MEND implicitly incorporates this knowledge and 
recognizes that there are three parts to recovery from pre¬ 
condition errors: undo certain actions, reachieve precondi¬ 
tion, and redo the undone steps. Knowledge about what 
needs to be undone and what needs to be redone is part of 
the failure reason model. For example, 

(UNDO-STEPS (PRECONDITION GRASP (ROCK) 

WITHIN-GRASP (ROCK)) OPEN) 

(REDO-STEPS (PRECONDITION GRASP (ROCK) 

WITHIN-GRASP (ROCK)) GRASP(ROCK)), 

says that if failure in GRASP is attributed to the precondition 
error of the object not being WITHIN-GRASP, the OPEN 
undoes the effect of GRASP, and that GRASPing the object 
needs to be redone after WITHIN-GRASP has been re¬ 
achieved. 

The situation is entirely similar for information errors, and 
recovery is achieved by undoing certain actions, getting the 
necessary information, and redoing the undone steps. Facts 
such as these are used in patching together a recovery plan 
in a rather simple and direct manner, after that the expla¬ 
nation of the failure has been determined. When multiple 
explanations remain even after analysis, MEND applies an 
ad hoc technique using severity codes to identify the more 
serious failure reason, in the hope that the recovery plan 
will work even if the explanation is wrong. 

PERSPECTIVE AND RELATED WORK 

Before summarizing the work described in this paper it 
would be worthwhile to consider related work and place this 
work in perspective. 

Several studies that have dealt with the problem of plan¬ 
ning have adopted the paradigm of devising a simple plan 
for achieving the given goal, identifying and characterizing 
the problems in such a plan, and finally correcting the prob¬ 
lems in the original plan. It is fairly clear that techniques 
used in systems adopting such a paradigm have applicability 
to the problem of error recovery. 

Sussman, 15 in his investigation of skill acquisition, char¬ 
acterizes failures in terms of domain-independent goal in¬ 
teractions such as precondition conflict between conjunctive 
subgoals, and the system corrects these by strategies such 
as reordering actions in the plan, introducing preparatory 
steps, etc. Sacerdoti 13 and Tate 16 use similar characteriza¬ 
tions of subgoal interactions in their planning systems, but 
the process by which the characterization is achieved has 
been improved and made more systematic. 

In MYCROFT, 6 violations between the actual performance 
and the intended line drawing described by a model are 
determined, and then used in correcting the plan. Similarly, 
Fahlman 1 uses domain-dependent stability analysis to iden¬ 
tify instabilities in blocks world structures, and corrects 
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these by the use of subassemblies, counter weights and 
scaffolding. 

We note the following points that identify the similarities 
and differences between MEND and the systems mentioned 
above. 

A. In a general sense all of these systems have the ca¬ 
pability of dealing with execution time failures. How¬ 
ever, Sacerdoti’s system NOAH is the only one of 
these that deals specifically with error recovery. In 
aiding an inexperienced human to correct an error, 
NOAH performs a hierarchical search for more de¬ 
tailed descriptions of the incorrectly performed task 
to identify the failure. 

In contrast to the above systems, MEND more di¬ 
rectly models why actions fail during execution, and 
characterizes failure in terms of operational errors, 
information errors, precondition errors, and constraint 
errors. This gives MEND the capability of handling 
real world problems such as inaccuracies in effectors 
and sensors. 

B. MEND uses a data structure called a failure tree to 
represent explanations of the failure. From a structural 
point of view the failure tree is domain independent, 
and reflects the interrelationships between different 
steps in a plan. Further, it allows domain-dependent 
details to be attached to nodes in the tree. This struc¬ 
ture should therefore have applicability in character¬ 
izing failures in many domains. 

C. MEND’s domain-independent scheme for failure rea¬ 
son analysis identifies the kind of reasoning necessary 
for analysis of execution time failures, and shows how 
this type of reasoning can be incorporated in a com¬ 
puter controlled system that performs tasks in the real 
world. 

D. It identifies the kind of knowledge about the domain 
necessary for performing failure reason analysis, and 
suggests how such knowledge can be incorporated in 
a failure reason model associated with each action. 

E. It further shows how knowledge about steps needed 
to correct failures can be used in conjunction with the 
results of failure reason analysis to effect recovery by 
means of very simple planning strategies. 

LIMITATION AND EXTENSIONS 

Failure reason analysis as it is implemented in MEND can 
certainly be improved and made more general by including 
subgoal interaction problems. However, MEND assumes 
that it has been given logically correct plans and does not 
need to consider problems arising either from incomplete 
knowledge about the world or from ignoring interactions in 
the first order solution to a problem. 

Another dimension for improvement in failure reason 
analysis is in handling mechanism failures. An extension in 
this direction is possible by using a representation of mech¬ 
anisms such as the one suggested by Rieger. 12 We could also 
imagine extending failure reason analysis to handle dynamic 
worlds, but such extensions are likely to be difficult since 


we know little about how to deal with activities of other 
agents who may or may not cooperate in the execution of 
tasks. 

An important problem which MEND does not directly 
address is that of new information that may invalidate sub¬ 
sequent steps in the plan. Though MEND will fumble along 
by correcting the failure as it occurs, it is short-sighted in 
not anticipating failures. Hayes 7 has dealt with this problem 
by means of a representation of plans which makes explicit 
the dependency between subgoals and decisions made in 
arriving at these subgoals. 

CONCLUSIONS 

The study described in this report has ihown that extending 
the modeling of actions to include a failure reason model 
allows a robot to address itself more directly to the problem 
of error recovery. The failure reason model provides a 
means of representing the knowledge about why actions fail, 
knowledge about their manifestation, and finally knowledge 
about what can be done to correct failures. 

A specialized technique for performing common sense 
reasoning, appropriate for planning recovery from failure, 
is embodied in failure reason analysis. This analysis uses 
the failure reason model to build a failure tree to represent 
possible explanations of the failure. Planning recovery from 
failures by determining the explanation of the failure has 
proven to be a direct and simple way of handling error 
recovery, at least in the simple manipulation tasks that have 
been considered. The effectiveness of this method in more 
complex problem domains is yet to be investigated and re¬ 
quires a more sophisticated planning ability than the one 
incorporated in MEND. 

Finally, it is important to note that error recovery through 
failure reason analysis is but one approach to the problem. 
Another effective approach based on the effects rather than 
on the causes of failure has also been investigated as part of 
this study, 12 and can supplement failure reason analysis in 
planning recovery from failures. 
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Design automation 


Over the last decade design automation has emerged as an irreplaceable tool 
for engineering design. The design of today’s complex systems such as aircraft, 
ships, spacecraft, chemical plants, automobiles and computer systems would be 
unthinkable without extensive computer aids for analysis, simulation and syn¬ 
thesis. 

Whereas in the past design automation tools consisted of a collection of loosely 
connected programs, the current trend is towards an integrated design automation 
system centered around a common design database. 

The last decade also witnessed the emergence of low-cost computer graphics 
terminals that made it possible to use man-machine interaction in a cost-effective 
way. 

Three sessions explore the various aspects of design automation during this 
conference. The first session deals with automated design methodologies for large 
hardware/software complexes. Digital systems have become increasingly com¬ 
plex and new design strategies are required in order to achieve satisfactory 
designs. The second session deals with the layout of very-large integrated circuits. 
Due to the explosive development of integrated circuit technology, we are now 
faced with the problem of efficiently laying out extremely large circuits. The 
papers in this session explore several ways of solving this problem in an effective 
manner. Finally, the third session discusses mature design automation systems. 
In this session we try to look back at some existing systems in order to evaluate 
their usefulness and impact. 
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IC design—Misery or magic 


by K. J. LOOSEMORE 

Pompador Ltd. 

Hertfordshire, England 

Ever since integrated circuit designers began to put thou¬ 
sands of transistors onto a single chip the cost, in terms of 
human labor, required to lay out the circuit has been ex¬ 
tremely high. If we analyze the layout task we find, in 
general, that two almost distinct phases are involved. First 
the designer will design the basic building blocks for the 
circuit and connect them together. The former of these two 
phases of design requires considerable expertise and expe¬ 
rience, making efficient automation extremely difficult. The 
second phase, however, only requires lower grade labor, 
since the efficiency of this part of the layout in terms of size 
and electrical characteristics is usually less critical. The 
second phase, none the less, requires a considerable pro¬ 
portion of the design time and it is this fact, together with 
the need for it to be carried out from scratch for each new 
design and its lower criticality factor that make it a candidate 
for automation. 

The MAGIC suite of programs has been developed at the 
UNIVERSITY of EDINBURGH both to address the inter¬ 
connection problem and to attempt to provide some struc¬ 
turing facilities for the more efficient design of integrated 
circuits. 

The system consists of several programs centered around 
a main automatic/interactive layout generator. It is this main 
layout generator which is of most interest but it is worth¬ 
while to give a general description of the rest of the suite in 
order to see how the system fits together. 

The designer begins by taking his circuit diagram and 
splitting it up into a number of functional modules (e.g., a 
stage of a shift register or part of a decoder). Specifications 
for these functional modules are then written in the sys¬ 
tem’s input language. At this stage there are only two re¬ 
quirements, first the rectangular outline of each module must 
be specified and secondly the positions of the interconnect 
points (called “pins”) to the modules, must be defined. It 
is not necessary at this point to have the logic internal to 
the module defined. This is required only at the final output 
stage of the suite. 

Bonding pads around the outside of the chip are designed 
in a similar manner, however, their positions around the 
chip are vaguely specified (only side allocations and order 
are required). These definitions, once produced, would nor¬ 
mally be kept in a library file (Refer to Figure 1) and made 
available for future designs. 


After specifying the logic modules the details of their 
interconnection are defined. For each connection net a list 
of connections is given and, in order to minimize errors at 
this point, it is also necessary to include a list of all defined 
connection points which for this particular design are not 
connected. This feature allows the program to detect acci¬ 
dental non-connections. 

In addition to the definition of the circuit the conductor 
width must be specified and also an estimate given for the 
final chip size. When the information is complete the input 
file and library file are compiled and an internal system file 
generated from which the layout program will proceed. Be¬ 
cause several attempts are normally made to iterate to the 
best layout, this precompilation phase removes the need for 
the data to be re-checked for every iteration. 

The program builds a chip layout in a series of successive 
levels working from one edge (bottom) of the chip progres¬ 
sively towards the opposite edge. At each level, an area 
(slot) at a time is locally optimized; components are chosen 
for each slot which are most closely connected to the exist¬ 
ing portion of the layout and to any other components al¬ 
ready selected for the current slot. Placement of components 
and routing of conductors takes place in parallel allowing a 
global view of the problem to be used and making it possible 
to organize the layout very efficiently. 

The program begins by taking the estimate of the chip size 
and assuming a square chip, calculates the side length of the 
chip. The bottom edge of the chip is then laid down and one 
of the groups of pads specified in the input file is assigned 
to this side. The set of pads is allocated by finding the 
component most closely connected to all the pads then 
choosing the set of pads most closely connected to this 
component. This means the first group of pads will, as near 
as possible, all have a target connection in the first slot thus 
reducing the amount of routing to successively higher slots 
until the connection is made. 

Once the first group of pads has been allocated, the other 
groups of pads are assigned to other sides of the chip in the 
order specified in the input language. 

Layout in earnest can now begin. The layout proceeds 
from this point in a series of rectangular slots, each defined 
by the lowest level reached in the already completed layout 
and limited horizontally by obstacles protruding up at either 
end of the horizontal plateau, (see Figure 2.) Obviously the 
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LIBRARY FILE: 

“COMP” INV(7,63) 

“PINS” 

1(0,14)M 
2(0,24)M 
3(4,63)M 
4(5,0)M; 

“RECT”(D 0,0:7,12; 

“RECT”(1) 0,15:7,13 
“RECT ”(1) 0,52:7,11; 

“GROUP”H (3,4) 1,1,0; 

“RECT”(3) 2,11:3,5; 

“GROUF’H (3,4) 1,22,0; 

“RECT”(3) 2,23:3,30; 

“GROUP”H (3,4) 1,55,0; 

“RECT”(5) 0,21:7,7; 

“RECT”(5) 0,10:7,7; 

“END” 

“COMP” NAND(7,70) 

“POWER” 2 
“PINS” 

4(4,70)M 

3(7,36)M 

1(7,27)M 

2(7,16)M 

5(4,0)M; 

“RECT”(1) 3,3:8,12; 

“RECT”(1)3,18:8,7; 

“RECT’Xl) 3,28:8,12; 

“RECT’XD 3,63:7,10; 

“GROUF’H (3,4) 4,4,0; 

“RECT”(3) 4,14:6,5; 

“RECT”(3) 4,24:6,5; 

“GROUP”H (3,4) 4,34,0; 

“RECT”(3) 5,35:3,29; 

“GROUF’H (3,4) 4,67,0; 

“RECT”(5) 3,66:7,7; 

“RECT”(5) 3,3:8,7; 

“RECT”(5) 3,13:8,7; 

“RECT”(5) 3,23:8,7; 

“RECT”(5) 3,33:8,7; 

“END” 

SOURCE FILE: 

“UTIL” 400 
“CONWID” 4 
“LIBCOMP” INVl(INV) 

“LIBCOMP” INV2(INV) 

“LIBCOMP” NANDl(NAND) 

“PADGROUP” 

“LIBPAD” PADl(PAD) 

“LIBPAD” GROUND(PAD) 

“END” 

“PADGROUP” 

“LIBPAD’ POWER(PAD) 

“END” 

“NODE” POWER,INV1(1),INV2(1),NAND1(1); 

“NODE” GROUND,INV2(2) IN V 1(2) ,NAND 1(2); 

“NODE” PAD 1 ,INV 1(3),INV2(4); 

“NODE” INV2(3),NAND1(5); 

“UNCON” INV 1(4),NAND1(3); 

“FINISH” 

Figure 1 

first slot is the width of the entire chip and successive slots 
are generated in much the same way that rectangular stones 
of varying size are laid when building a stone wall. 

An attempt is made to optimize the use of the area in the 


slot both by choosing the best set of components to fit the 
area of the slot and by minimizing the amount of intercon¬ 
nection required. In filling the slot with components it is 
possible to calculate the total width required for a given set 
of components. This is simply the sum of the widths of the 
components (with appropriate orientations) and the width 
required for routing conductors vertically between the com¬ 
ponents. This latter factor consists of the space required for 
conductors going straight up through the slot without con¬ 
necting to anything and the space required for connections 
to side pins of components. If the order and orientation of 
the components is well chosen this space requried for ver¬ 
tical routing can be minimized. For instance if a net connects 
to two or more pins in the slot there are two possibilities; 
(1) the pins are in different intercomponent gaps, in which 
case two or more vertical conductors are required and (2) 
the pins are in the same gap, in which case only one vertical 
conductor is required to service both of them (see Figure 
3). 

There are several other features which influence the ori¬ 
entation and ordering of components in a slot. Firstly, it is 
more efficient, in general, to have components oriented with 
their largest sides horizontal. This tends to produce a few 
large plateaus on which to build further slots rather than 
several small ones thus increasing effective use of area due 
to reduced fragmentation. Secondly the orientation of com¬ 
ponents is a critical factor in determining the number of 
conductor cross-overs required. For electrical reasons it is 
important to have as few cross-overs as possible so, within 
the constraints set by the previous requirements the com¬ 
ponents is a critical factor in determining the number of 
cross-overs in their connections with the already existing 
part of the layout. The third factor on which placement 
depends is the ordering of the components in the slot. 

An attempt is made to minimize the horizontal routing 
required in the slot by taking this into consideration when 
ordering and orientating the components. There is obviously 
a trade-off between reducing the space requried for vertical 
routing and reducing the amount of horizontal routing since 
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in order to put two pins in the same intercomponent gap it 
may be necessary to move one of the components further 
away from its best horizontal position and increase the 
amount of horizontal routing required. 

These processes will have reduced the amount of space 
required for the current set of components as far as possible. 
If there is some extra space left in the slot an attempt is now 
made to find the component which will best use that space. 
When this operation has been completed the slot will usually 
be very tightly packed with little unused space. However 
any unused space is now inserted next to the lowest com¬ 
ponent in the slot in order to increase the width of what will 
probably turn out to be the next slot. The x coordinates of 
the components are now fixed allowing the x coordinates 
of connection targets to be fixed so that horizontal routing 
can take place. The y coordinates of the components are 
left undefined and are only fixed as each component has all 
routes which need to pass beneath it completed. The com¬ 
ponent can then be allowed to settle on top of the horizontal 
routing under it. This delayed fixing of y coordinates is 
necessary because it is extremely difficult to calculate the 
amount of space required for routing under a component 
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without actually doing it. The next part of the operation is 
to perform the horizontal routing in the slot. Because all the 
coordinates have been fixed it is possible to route horizontal 
segments even though the exact positions of the components 
are not known. For this routing a system of priorities is 
used. If there are two or more types entering the slot from 
below that need to be connected, these are given highest 
priority so that they are connected as early as ^possible and 
do not interfere with the actual intercomponent routing. 
Next highest priority routes are those which need to pass 
under components. The effect of completing these is that 
the components may be placed as low as possible in the slot 
and take less height. Lowest priority routes are those that 
only pass from the bottom of the slot to the top, and routes 
from component sides to the top of the slot. The algorithm 
operates by performing horizontal routing at increments of 
one conductor pitch from the bottom of the slot. As many 
routes and potential routes are completed at each routing 
level and when no further routing is possible vertical con¬ 
nections are made from the lower ends up to the next level 
and a fresh attempt is made there. As all routes below each 
component are completed, the component position is fixed 
and vertical routing to it is made. In parallel with this activity 
routes going to the top of the slot for connection in later 
slots are made and when all the components are fixed and 
the partial routing completed the processing of the slot is 
finished. This process continues for all the slots until even¬ 
tually there will be no more components left to place and all 
the loose conductor ends will be joined in the final slot. As 
soon as the layout is finished all the pads are added and an 
output file is produced containing the details of the layout. 

This fully automatic system, when tested on a manually 
implemented design produced a chip about 1.25 times the 
linear dimension of the manual design. It was found that 
there were several instances of awkwardly placed compo¬ 
nents or densely connected components where the program 
did not produce an optimum layout. In order to improve the 
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situation an interactive phase has been added which allows 
the program and the user to work together. A menu con¬ 
taining the names of all the components in the design is 
displayed on the right hand side of the display screen on 
which the design is drawn (see Figure 4). At the beginning 
of each slot the user is given the chance to choose the 
component positions and orientations. He may either fill the 
slot or break off on a partially full slot and instruct the 
program to fill the rest of the slot automatically or to leave 
the rest of the slot vacant. Using this facility the program 
improved on the completely automatic technique giving a 
result about 1.15 times the linear dimension of the manual 
layout. 

It is worth pointing out the reasons the program does not 
equal or improve upon the layout produced manually. There 
are two main reasons. The main one is that the manual 
designer often designs his components to butt together. The 
program is unable to detect this fact and will always leave 
space for, and perform, some routing between the compo¬ 
nents. Research is under way to test the feasibility of pro¬ 
viding some method to butt together some components au¬ 
tomatically. In a completely “standard cell” 1 system a 
different algorithm is used which takes advantage of all the 
aspects of “standard cells”. The second reason why the 
automatic layout is larger is that designers often generate 
special versions of modules to fit into local contexts. This 
has the effect of reducing the amount of connection required 
had the “standard” version been used. The program, having 
no knowledge of what is contained within components, is 
unable to adjust component sizes or shapes and must make 
do with what is provided. For small numbers of custom 
designed IC’s the design time/chip size trade-off is a good 
one and will become better for even larger numbers of pro¬ 
duction units when the new VLSI processes become com¬ 
monplace and space ceases to be critical. 

POST PROCESSORS 

The interactive design program produces technology in¬ 
dependent output assuming there are two layers available 
for routing. In order to realize the circuit in the required 
technology an appropriate post processor is required. Typ¬ 
ically the post processor takes the design file produced by 
the layout program and assigns conductor segments to lay¬ 
ers, inserts underpasses where necessary and locates con¬ 
tact windows where interlayer connections are required. It 
is at this post processor stage that we need to ask the 
question “What is going to happen to the design from 
here?” At Edinburgh the design is passed to an independant 
suite of drafting programs collectively known as GAELIC. 2 
The post processor searches the source and library files for 
the component definitions and extracts from them the details 
of the internal logic specified in GAELIC manual input lan¬ 


guage. This is output to a file together with all the intercon¬ 
nections and contact windows in the GAELIC manual input 
language, thus presenting the GAELIC system with a com¬ 
plete design which looks as though it were produced man¬ 
ually. Using special programs in the GAELIC suite it is 
possible to automatically generate drive tapes for pattern 
generators and other mask-making devices. The use of the 
MAGIC suite of programs in conjunction with GAELIC 
substantially reduces the time required to design an inte¬ 
grated circuit. Used properly, the system can reduce design 
time by a factor of ten. This is achieved by storing in the 
library the complete component specification from the pre¬ 
vious circuits; as the library is built up, all that is necessary 
to perform a new design is to select appropriate items from 
the library and to specify the interconnections. After this 
stage everything can be automatic. The chip is laid out by 
MAGIC and the appropriate GAELIC post processor is used 
to generate drive tapes for a mask making device. If any 
interaction is required, the facilities exist, not only in 
MAGIC but also in terms of the substantial graphic editing 
facilities in GAELIC. 


APPLICATION CONTEXT 

The MAGIC suite of programs is applicable to all areas 
of IC design in which modularity of design exists. Both the 
rectangular components described earlier and standard cells 
can be laid out, although the latter layout is not as efficient 
as if a specialized standard cell program had been used. 

The system is most effective when used for irregular de¬ 
signs where patterns are not repeated a larger number of 
times such as in memory chip designs. In the latter instance, 
the designer can often produce an optimal circuit design in 
such a short time that the overhead represented by the extra 
space requirement using MAGIC is not warranted. The sys¬ 
tem is applicable to all current IC technologies since suffi¬ 
cient parametrization exists for the details of individual tech¬ 
nologies to be included in the design specification. The use 
of post processors, one for each technology, allows the 
design to be optimized for each particular technology auto¬ 
matically. 

As VLSI techniques become more commonplace the dif¬ 
ficulty of laying out a chip by hand is going to be so extreme 
that unless systems like MAGIC are used, the full potential 
of VLSI is not going to be realized. 
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STICKS—A graphical compiler for high level LSI design 

by JOHN D. WILLIAMS 
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INTRODUCTION 

The need for good design automation in the area of inte¬ 
grated circuit layout is severe. It is believed that the STICKS 
system does much to fill that need. STICKS is a computer 
aided design system which frees the designer from the te¬ 
dious aspects of IC design and allows him to concentrate on 
the more creative and necessarily human side of the design 
process. With STICKS the designer is allowed to divorce 
himself from the usual precise sizes, spacings, and interre¬ 
lationships required for an IC layout and, instead, submit 
his creativity in the form of rough, freehand sketches. The 
computer makes whatever adjustments are necessary in 
order to generate an error-free layout. The opportunity is 
then given to verify and, if necessary, modify the computer’s 
interpretation of the design. An interactive loop is thus 
formed with the computer doing all of the tedious, detailed 
work and the human designer providing the necessary in¬ 
spiration. 

Using traditional methods, large scale integrated circuit 
layout is a tedious, time consuming and error prone process. 
IC layout is a procedure which results in the placement and 
interconnection of each of the thousands of components 
which form the integrated circuit. The designer accom¬ 
plishes this by drawing several superimposed “mask” layers 
using a drafting board which together, control the areas 
where various chemical diffusions and etchings will be made 
on the silicon wafer that forms the IC. Components such as 
transistors, diodes, resistors, and their interconnections are 
formed by various combinations of these masked areas. 
Unfortunately, the specific sizes of and spacings between 
components is critical, preventing the designer’s job from 
being an easy one. 

There is an overall goal in IC design to pack as much 
circuitry as possible into a small area. The more circuitry a 
particular IC contains, the more valuable it is and, conse¬ 
quently, the more marketable it is. The physical size of the 
chip, however, is a major factor constraining the number of 
components. Due to a parameter known as defect density, 
the larger the chip, the less likely it is to work and the more 
it will cost. Defect density is the number of random material 
defects per given area which result both from the starting 
silicon material and from normal losses in the manufacturing 
process. The solution, one would think, would be to simply 
make everything smaller thereby increasing component den¬ 


sity. Other limitations in the capabilities of the process, 
however, make this solution impossible. As the components 
get smaller, the more effect small random defects will have 
and, again, the less likely the circuits will work. 

The absolute minimum sizes and spacings in a layout are 
embodied in a set of “design rules” which are specific to 
the particular manufacturing process. They specify in detail 
the relationships between any two components or, more 
fundamentally, mask level rectangles. Only when all of the 
design rules have been met will the circuit be guaranteed to 
be manufacturable. 

It should be understood that the design rules are not 
necessarily simple width and spacing definitions, they are 
often dependent upon complex relationships between the 
mask rectangles and the components intended to be imple¬ 
mented. The number and complexity of the design rules are 
a major contributor to the difficulties and tedium encoun¬ 
tered in the layout process. Each mask rectangle is con¬ 
strained in two dimensions by several other rectangles on 
the various mask levels. Indirectly, the placement of every 
rectangle in every mask is dependent upon the placement of 
most, if not all, of the rectangles in the entire IC. 

The designer approached the generation of the layout by 
placing the first element and then, sequentially, building 
elements around it, while trying to leave just enough room 
for future elements. This is fairly difficult because it is hard 
to know just how future elements will fit together. The 
designer generally copes with this problem by adopting a 
trial and error approach. When the rectangle he is trying to 
place will not fit without violating a design rule, he must 
backtrack far enough to correct the problem. This is accom¬ 
plished by erasing and repositioning any previously placed 
rectangles which led to the difficulty. Figure 1 shows a 
simple example of this situation. This particular example is 
not a serious one but, in actual practice, several hundred 
structures may have to be erased and redrawn each time a 
problem of this type occurs. It is virtually impossible for 
even a highly experienced designer to be able to look ahead 
far enough in order to avoid having to backtrack. 

Traditional layout methods pose additional problems be¬ 
sides those already mentioned. The number and complexity 
of the design rules make a certain number of design rule 
violations inevitable. In fact, we have found that approxi¬ 
mately ten undetected design rule violations are made for 
every thousand hand drawn rectangles. This number rep- 
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Figure 1 


resents only those errors which are found after the layout 
has been completed. These can have disastrous effects if 
global design changes must be made to correct them. The 
problem is that the designer has packed all of the mask 
rectangles as close as possible together (within design rule 
constraints) in an effort to maximize circuit density. If a 
design rule violation is found, it means that the structures 
were packed too closely together, implying that they must 
be moved apart slightly. Unfortunately, moving these struc¬ 
tures apart would cause additional design rule violations 
with surrounding structures. The corrections for these errors 
tend to ripple out to the edge of the drawing, forcing the 
repositioning of many structures. Since the errors were 
found only after the entire layout had been completed, thou¬ 
sands of structures could be involved. Then, inevitably, the 
process of physically repositioning all of these elements 
could easily produce more design rule violations causing the 
problem to repeat. 

The need to reposition large numbers of circuit elements 
can also occur when actual circuit modifications must be 
made such as when logic errors are found after the IC has 
been fabricated for the first time. This has the same effect 
as the discovery of a design rule error. Once the layout has 
been created, it is often very difficult to add any additional 
circuit elements without moving a lot of rectangles. Again, 
changes like this are always fraught with the danger of in¬ 
troducing new errors. 

With these difficulties in mind, it is easy to understand 
why layout represents a very costly stage in IC develop¬ 
ment, both in terms of time and effort. 


THE STICK DIAGRAM 

The designer’s input to the STICKS system, called a stick 
diagram, is a high level, schematic-like representation of the 
intended circuit. Circuit elements like transistors, diodes. 


and contacts (vias) are represented as single symbols instead 
of as complex sets of overlapping mask levels. Likewise, 
silicon and metal interconnects are represented by their cen¬ 
terlines. A stick diagram is unlike a schematic, however, in 
that it includes rough relative positional information in ad¬ 
dition to the usual connectivity information. An example of 
a stick diagram is shown in Figure 2. Like a schematic, the 
actual distances between the elements in this drawing are 
completely arbitrary. What is important however, is the 
relative position between the elements. The fact that a tran¬ 
sistor, for example, is “above” a metal interconnect or 
“somewhere to the left of’ a particular diode are the prop¬ 
erties which make a given stick diagram unique. 

The lack of actual geometrical spacing information is an 
important characteristic of the stick diagram. In using the 
stick diagram as a design “language”, the IC designer need 
not worry about the actual location of the circuit elements 
he places. As a result, his design is nearly as easy to draw, 
modify, and interpret as a schematic. In fact, it has been 
found that many designers bypass the schematic entirely 
and directly translate their logic diagrams into stick dia¬ 
grams. The use of single symbols for complex structures 
and centerlines for interconnects as well as the complete 
absence of specific inter-element spacings makes the stick 
diagram very fast and easy to draw. The designer does not 
have to worry about leaving enough room for surrounding 
structures, there is always room to add them. Unlike designs 
produced at the mask rectangle level, modifications can be 
made on a stick drawing without the need for accompanying 
major global rework. If it is necessary to add a structure, 
one can always “find” room by simply drawing it smaller 
and inserting it where desired. Likewise, deleting a structure 
does not leave a hole since the spacing between elements in 
a stick drawing is arbitrary. 

This arbitrary spacing also means that the designer is 
removed from the specifics of the particular manufacturing 
process. He does not have to know what design rules are or 
where they are applied, nor does he have to know that mask 
levels like nitride, implant, diffusion, and oversize gate even 
exist (e.g., a transistor is a single symbol instead of a com¬ 
plex system of precisely aligned, overlapping masks). 

The difference between a schematic and a stick diagram 
(the addition of relative positional information) makes the 
translation of the stick diagram to a full mask layout a fairly 
straightforward process. Without this added information, 
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the translation process would have to include component 
placement; a problem with which computers traditionally 
have had a difficult time. 

The stick diagram is a happy medium between the sche¬ 
matic and the full mask layout. It is at a high enough level 
to allow the designer to concentrate primarily on the creative 
aspects of the design yet at a low enough level to completely 
describe the final geometries in such a way that translation 
to full mask rectangles can be automated without loss in 
layout quality. 

IC design at the stick diagram level utilizes the capabilities 
of both man and machine in a very effective way. Humans 
have very good global perspective; they can look at a draw¬ 
ing and immediately identify how elements should roughly 
fit together to maximize density. They are very poor, how¬ 
ever, at identifying all of the constraints on each element of 
the drawing and precisely aligning them so that no design 
rules are violated. The computer, on the other hand, is 
extremely good at calculating local element separations but 
poor at making global decisions. By allowing the human to 
design with stick diagrams and having the machine translate 
these drawings into full layouts, each is doing what he does 
best. The human is using his experience and global insight 
to create the overall plan for the layout while the computer 
is left to do the tedious, detailed work that it does so well. 

In addition to transistors, contacts, and interconnects, the 
stick diagram may also include mask rectangles and cells 
(previously designed layouts). 

There are two different types of cells available to the 
designer. The first type, called a “macro,” is simply a stick 
diagram which has been previously stored in a library for 
future use. When a macro is placed in a stick diagram, it 
acts as though it had been drawn by hand in its assigned 
location. It loses any identity it may have once had and, 
upon subsequent translation to mask layout, is free to 
stretch, shrink, and mold to its environment depending upon 
the influenc of surrounding structures. The value of macro 
cells lies in the ability to pre-define small, commonly used 
sub-circuits such as gates and flip-flops, eliminating the need 
for the designer to repeatedly draw them. 

The other kind of cell is called a black box cell. It is 
created by allowing STICKS to generate a layout from a 
stick drawing (which may, incidentally, contain other black 
box cells) and then freezing this layout by declaring it to be 
a new black box cell. This cell may be used in subsequent 
stick drawings as if it were a logic symbol. It is an outline 
of the actual layout with labelled ports signifying places 
where external circuitry may be connected. The cell is also 
labelled to denote its function. The use of black box cells 
allows the designer to operate at a higher level than that of 
transitors and diodes. He can define functional blocks of 
circuitry and use these in future stick diagrams. 


THE STICKS COMPILER 

The job of the STICKS compiler is to translate the de¬ 
signer’s stick diagram into a complete mask layout which is 


consistent with design rules. It performs this operation in 
two main stages. The first stage (called the spacing algo¬ 
rithm) repositions each of the elements in the stick diagram 
in a way which brings them together as close as possible 
without violating design rules. The second stage (called rec¬ 
tangle generation) converts the resulting “spaced” stick dia¬ 
gram into a full mask layout. 

The spacing algorithm’s ability to function properly is due 
to the fundamental nature of the stick diagram. The stick 
diagram is composed of relocatable elements wired together 
by stretchable interconnects. It is this stretchability which 
provides the added degrees of freedom that are necessary 
to create an error free design. As long as each element is 
free to move and any connections to any other elements are 
infinitely stretchable, a valid layout can be produced. This 
is true because of the way IC design rules are defined. 
Design rules prevent any two elements from being too close 
to each other; all design rules are specified as a minimum 
distance. This implies that a perfectly valid layout could be 
produced from a stick diagram with the above properties 
simply by stretching all of the interconnects so that elements 
are infinitely far apart from each other. Opposing this phi¬ 
losophy, however, is the desire to maintain the highest pos¬ 
sible layout density. The solution then, is to move the ele¬ 
ments only as far apart as necessary in order to not violate 
any design rules. This, in theory, will create the smallest 
possible valid layout. 

The problem, then, is to know in which direction and how 
far to move each element to produce the optimum layout. 
The approach the STICKS compiler takes is to divide this 
two-dimensional problem into two one-dimensional prob¬ 
lems. This is accomplished by treating the horizontal struc¬ 
tures separately from the vertical structures. In doing so, 
the algorithm only has to worry about moving each element 
in one direction at a time. Basically, the spacing algorithm 
begins by placing all structures which lie on the bottommost 
horizontal coordinate at the bottom (origin) of a blank work 
area. The next step is to attempt to place the structures 
which are on the second horizontal coordinate as close as 
possible to this origin. This is accomplished by looking up 
the design rules between the various structures on the two 
coordinates and deciding where each second coordinate 
structure must be in order to be as close as possible to the 
first coordinate structures without violating design rules. 
The second coordinate structures are then actually moved 
to these new locations and assigned an actual distance from 
the origin. The compiler has done two things: it has moved 
structures relative to each other in a way which tends to 
increase circuit density and it has also assigned an actual 
physical vertical spacing between structures. 

It should be realized that whenever a horizontal structure 
is moved to a different coordinate, any vertical interconnec¬ 
tions which are attached must be stretched accordingly. This 
maintains the connectivity of the circuit. 

The horizontal pass continues with each subsequent co¬ 
ordinate of horizontal structures moved down as far as de¬ 
sign rules permit. When all of the horizontal structures have 
been spaced, the spacing algorithm is used in the opposite 
direction. In this pass, the vertical structures are reposi- 
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tioned while any horizontal interconnects are appropriately 
stretched. 

As soon as both passes have been completed, the designer 
is given the ability to perform optimization on the stick 
diagram with the help of the compiler. These techniques will 
be discussed briefly in the next section. Once the designer 
is satisfied with the density of the spaced stick diagram, he 
can ask the compiler to go into its second stage of operation: 
rectangle generation. 

As its name implies, rectangle generation takes the stick 
diagram which has been spaced to design rule specifications 
and builds the corresponding mask rectangles. It does this 
by expanding the interconnect centerlines to their full width 
and by generating all of the mask rectangles associated with 
the circuit elements. It also recalls the contents of black box 
cells from the system library and substitutes the appropriate 
set of mask rectangles for the cell outline found in the stick 
diagram. There is no danger of a design rule violation oc¬ 
curring as a result of this procedure since the spacing algo¬ 
rithm took all of this into account when it performed its 
operation. The product of rectangle generation is a complete 
mask layout ready to be sent to mask making equipment. 
Figure 3 shows the results of each stage of the compilation 
of a simple stick diagram. 

One section of the compiler (called the knowledge base) 
consists of a set of production rules in which are coded the 
design rules between any two structures. Whenever the 
compiler is trying to decide how far apart two given struc¬ 
tures must be, it consults its knowledge base which, in turn, 
analyzes the properties of the two structures and reports 
back the minimum separation distance. Besides the basic 
design rules, the production rules include additional knowl¬ 
edge which might be called layout heuristics. Layout heu¬ 
ristics are the designer’s “tricks of the trade” which he has 
developed through experience. One example of this would 
be the fact that a design rule which specifies the minimum 
separation between two metal interconnects can be ignored 
if the two structures are electrically connected. In fact, it 
might be advantageous to completely merge these two struc¬ 
tures thus saving area. STICKS has the capability to identify 
situations like this and make appropriate decisions. 

There are several advantages in basing the implementation 
of the STICKS compiler on the use of production rules. 
First, the complexity and tremendous number of relation¬ 
ships between IC structures make it very difficult to develop 
a traditional algorithm which is capable of performing the 
required tasks. The IC design rules as well as layout heuris¬ 
tics already exist in the designer’s mind as a set of rules 
which are selectively applied whenever the proper circum¬ 
stances exist. It is natural then, for a computer program to 
incorporate the same set of rules as its knowledge base. A 
second advantage is that it is impossible to incorporate all 
knowledge of IC layout the first time. In fact, it is almost 
certain that a continuous process of “tweaking” the per¬ 
formance of the system by modifying the knowledge base 
will exist throughout its usable life. Having to continually 
rewrite or patch an algorithm is not a pleasant prospect. 
Modifying a production rule based system, however, means 
only adding, deleting, or modifying the appropriate rules. 


Since the rules are virtually independent of each other, this 
can be done without adverse consequences. 

DESIGNER INTERACTION WITH STICKS 

An important feature of the STICKS system is that the 
designer has total control over the quality of the final layout. 
This control is maintained through several interactive func¬ 
tions which may be invoked after the compiler has per¬ 
formed its spacing algorithm on the entire drawing in both 
the horizontal and vertical directions. At this point, there 
are no design rule violations and all structures have been 
packed toward the lower left comer of the drawing. 

It is likely that this spaced stick diagram will not be as 
dense as it could be. Although each structure in the drawing 
will have been positioned as close to its adjacent structures 
as design mles permit, it is quite possible that unused areas 
exist. This stems from the fact that the spacing algorithm 
really is not looking at the global picture. It is assigning 
minimum spacings between adjacent components but cannot 
identify the situations when overall density might be im¬ 
proved if a particular local spacing were not set to minimum. 
This occurs most frequently when compression in the hori¬ 
zontal direction blocks subsequent compression in the ver¬ 
tical direction. Figure 4 shows a very simple example of this 
situation. In this example, a much more compact layout 
would have been obtained if the first group of horizontal 
structures in the center of the drawing had not been moved 
all the way down to the origin but, instead had been left in 
its original location. The compiler had no way of knowing 
this, since it just tries to pack everything toward the bottom 
left of the drawing. STICKS depends upon the designer to 
spot problem situations like this and allows him to remedy 
them through the use of a procedure called local optimiza¬ 
tion. 

Local optimization allows the designer to draw a boundary 
around any area in the spaced stick diagram and specify a 
direction in which structures that are included in the bound¬ 
ary are to be respaced. Figure 5 shows how this operation 
would work to fix the problem in Figure 4. It is done in two 
steps. First the horizontal group of structures which were 
originally moved down to the origin are directed to move 
back up and then the displaced vertical structures are di¬ 
rected to move to the left to fill in the unused area. There 
is no danger of design rule violations occurring as a result 
of this procedure since the spacing algorithm always does 
the actual repositioning of structures. The designer is merely 
giving it guidance from his global considerations. 

A second means by which the designer may interact with 
the system is through the use of fractures. Figure 6 is a 
before and after picture showing how fractures can be used 
to increase the density of a spaced stick drawing. The overall 
effect of fractures is to allow sections of circuitry which 
were originally constrained to a particular location by 
straight interconnects to fill in large unused areas of the 
layout. The fracture breaks a straight interconnect and al¬ 
lows it to jog in either direction at that point. The designer 
is actually giving his drawing an added degree of flexibility 
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in certain select areas, thereby giving the STICKS compiler 
freedom to do a better job. 

Another problem which can occur involves groups of cir¬ 
cuit elements which the designer would like to physically 
keep together in the layout. It is desirable, for example, to 
keep all of the flip-flops in a register together in a single 
row. This is especially important for circuit debugging pur¬ 
poses in the event that the output state of the register must 
be probed. It would be awkward to search all over the IC 
for all of the pieces Pf the register in order to identify its 
output nodes. This, however, might very well be the case 
if the spacing algorithm were left unchecked. It has no way 
of knowing which functional groups of elements should re¬ 
main together. To prevent the dispersion of such functional 
groups, the designer is given the ability to enclose all of the 
elements of the group in a rectangular perimeter called a 
corral. The corral has the ability to change absolute size; 
however nothing which is enclosed in it may leave its bound¬ 
ary. This effectively separates those structures from all oth¬ 
ers in the drawing. 

The final way in which the designer may interact with the 
system is by specifying the absolute micron distance be¬ 
tween any two structures in the drawing. This effectively 
limits the freedom which STICKS has to move structures 
around, giving the designer considerable control over the 
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final shape and size of the layout. Of course, there are perils 
associated with this ability. It is easy to over-constrain a 
drawing, especially when an absolute constraint imposed by 
the designer violates design rules. STICKS can identify 
these situations and inform the designer of the impossibility 
of his restrictions. 

Together, these interactive optimization techniques allow 
the designer to maintain ultimate control over the quality of 
his design. This control is believed to be essential in a design 
automation system of this kind. 


CONCLUSION 

The STICKS system offers the integrated circuit designer 
many advantages over traditional layout methods. He de¬ 
signs in a high level, graphical language which is very easy 
to draw, modify, and maintain. He no longer must worry 
about design rule violations since it is impossible for him to 
make any. The layout is built to design rule specifications. 

Additionally, the stick diagram possesses a certain amount 
of process independence. This means that once an IC is 
designed in the form of a set of stick diagrams, it may be 


“recompiled” into different mask layouts which are com¬ 
patible with other manufacturing processes. This feature 
helps to prevent the obsolescence of expensive IC designs. 
It is especially advantageous as the original process evolves. 
The designs can track the process by being recompiled using 
each new set of design rules. 

A final advantage is related to the ease with which the 
designer can produce a layout. Traditional layout methods 
are so tedious that even when it is discovered that the layout 
is poor with respect to circuit density, it is often just too 
much trouble to do anything about it. With STICKS, on the 
other hand, the designer is encouraged to try several differ¬ 
ent layout approaches and try to optimize for density. As a 
result, it is quite possible to obtain even higher quality lay¬ 
outs with STICKS than with a layout done completely by 
hand. 

Together, these advantages make STICKS an exciting 
solution to the difficult IC mask layout problem. 

REFERENCE 

1. Williams, John D., “STICKS—A New Approach To LSI Design,” Mas¬ 
ter’s Thesis, Massachusetts Institute of Technology, June 1977. 





A method for the automatic wiring of LSI chips 


by NING NAN and MICHAEL FEUER 
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INTRODUCTION 

The best known technique for finding a connection between 
two points is the Moore-Lee algorithm. 1 It is general and 
thorough, but leads to a long running time. Hightower 2 de¬ 
vised a line-probe wiring scheme which is much more effi¬ 
cient in finding such paths. Variants of this algorithm are 
probably the most popular means of wiring printed circuit 
boards at present. However, both of the above techniques 
suffer from a tendency for the last connections attempted to 
be blocked unnecessarily by previously wired ones. A more 
tentative approach to these connections was developed by 
Hitchcock, 3 whose cellular mazerunner specified connec¬ 
tions through cells without fixing their exact location until 
the end. Schemes which we call line packers for assigning 
routes to parallel channels have been devised by Hashimoto 
and Stevens 4 and Kemighan, Schweikert and Persky. 5 

The method described in this paper was motivated by a 
similar desire to defer final channel selection as long as 
possible and to give each connnection equal treatment. The 
method consists of subdividing the wiring space into blocks 
in a hierarchical fashion, assigning connections to block 
boundaries by perturbing a global trial solution, mapping the 
solution to a finer grid and eventually allocating wire seg¬ 
ments to channels in the vertical and horizontal directions 
by linepacking techniques. 

The results of an implementation of this method, using a 
two level hierarchy, have proven very successful in produc¬ 
tion use at IBM. 6 

THE WIRING PROBLEM 

The integrated circuit chip consists of a planar array of 
devices whose logic terminals (pins) are to be interconnected 
by metal according to a functional specification. The metal 
lines are made on two layers separated by an insulating 
material. In order to maximize wiring efficiency, the hori¬ 
zontal segments of a connection are made primarily on one 
plane (the horizontal plane) and the vertical segments pri¬ 
marily on the other (the vertical plane). The wiring channels 
on these planes are organized into horizontal streets and 
vertical avenues (Figure 1). 

In this paper, a net is taken to mean a set of pins which 
are to be connected by metal wires or those wires them¬ 


selves. A connection is a pair of pins which are to be inter¬ 
connected, or the metal path between these pins. A segment 
is a portion of a net which exists within one street or avenue. 

Changing planes can he accomplished by introducing via 
holes in the insulator between the two planes. These vias 
are made as needed and are not limited to the fixed patterns 
typical of printed circuit boards. 

HIERARCHICAL GLOBAL WIRING 

In the first phase of wiring, the chip is divided into a 
rectangular array of blocks each of which contains many 
wiring channels and circuit pins. A straight line segment 
separating adjacent blocks is called an edge. The function 
of global wiring is to decide which edges the global nets will 
cross. Nets whose pins are contained wholly within one 
block (local nets) are not routed at this phase. For large 
chips, each block can be subdivided after one stage of global 
wiring into smaller blocks, and the global solution can be 
further refined (Figure 2). 

Each edge can accommodate a number of crossing wires. 
This is called the channel capacity of the edge and is largely 
determined by the number of wiring channels crossed by the 
edge. Each block is assigned an internal capacity reflecting 
the ease of wiring past the internal nets or other obstacles 
within the block. The goal of global wiring is to avoid wiring 
more nets through blocks or edges than permitted by the 
available capacities. 

PERTUBRATION OF A TRIAL SOLUTION 

The global wiring can be done, in principle, by any of the 
traditional mazerunning or line-probe techniques. An essen¬ 
tial advance, however, was made by K. A. Chen 6 and later 
by J. Lee, 7 which we will call wiring by perturbation from 
a trial solution. The initial trial solution is built up by wiring 
each net indepenently. This results in an invalid solution 
where some regions are over-congested in the sense that 
wiring demand exceeds edge and block capacity. Nonethe¬ 
less, this initial solution resembles the real solution much 
more closely than does a blank wiring image. In addition, 
the initial solution has the satisfying property of treating all 
the nets equally. 
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The net configurations, which can be used to build the 
initial solutiin, vary widely: minimum spanning trees, Stei¬ 
ner trees, biased Steiner trees, chains, source of sink trees, 
and redundant graphs, such as J. Lee’s enclosing grid (Fig¬ 
ure 3). 

The perturbation consists of moving nets or segments 
away from congestion peaks. This rerouting can be accom¬ 
plished by mazerunning, parallel translation of segments or 
by the deletion of redundant segments. In each case, a net 
or segment is identified as crossing a congested region and 
then rerouted in a less congested area at the possible ex¬ 
pense of increasing its length. 

An illustration of this global wiring procedure on an array 
of 3x3 blocks is given in Figures 4-8. Each number in 
Figure 4 represents a pin of a circuit. The same numbers on 
the chip are to be connected to form nets. All the nets are 
connected in a Steiner fashion without reference to each 
other. The initial global solution in Figure 5 is, in general, 
over congested at some boundaries. The number of nets 
crossing from one block to another is defined as the channel 
demand at the boundary between the two blocks (Figure 6). 
Where channel demand exceeds capacity, as outlined in 
Figure 7, there is clearly a need for rerouting some nets. 

The reduction of congestion starts at the worst bounda¬ 
ries. Each net crossing the congested boundary is assigned 
a weight, which is a function of the congestion in the whole 


path of the net. The net with the highest weight is chosen 
first for rerouting. Since only improved paths are accepted 
in the process, the overall convergence is guaranteed. Figure 
8 shows a balanced solution after rerouting. 

The process can be visualized as the leveling of hilly 
terrain by pushing material from hill tops down the slopes. 
This results in lower and smoother contours with a small 
increase in total volume of material (net length). 


PIN SELECTION 

During the last stages of global wiring or the early stages 
of detailed wiring, it is advantageous to reassign nets to 
pins. The pins to which the nets must attach may often be 
permuted in small sets without any effect on logical function. 
For example, the input pins of a NAND or a NOR circuit 
are all equivalent. The exact choice of these pins has an 
effect on the number of channels which are available through 
a block, as illustrated in Figure 9. The objective of the pin 
selection analysis is to minimize the blockage of the wiring 
sources. 


NET SEGMENTATION 

After global wiring, the nets are assigned to certain hori¬ 
zontal wiring streets and vertical avenues for detailed wiring. 
Each street will contain a number of parallel segments which 
must be assigned to available channels. The wiring within 
each street or avenue is dependent on that in all the others. 
In particular, where a street crosses an avenue, the vertical 
and horizontal wiring are coupled. Still, this coupling is 
generally not strong enough to force the detailed wiring to 
use more channels than would be required for an uncoupled 
solution. 

Figure 10 illustrates the segmentation of a net N, in the 
vertical direction. Xj’s are the net points and aj’s represent 
the assignment points. After the vertical assignment process, 
net N is segmented into four independent subnets n t , n 2 , n 3 
and n 4 . 

Figure 11 shows the horizontal segmentation of the same 
net N. It should be pointed out that only one segmentation 
is required for a wiring procedure. 




Figure 2—Block hierarchy 


CHANNEL ALLOCATION 

The most constrained streets or avenues are wired first. 
The problem consists of a set of parallel segments to be 
assigned to channels. The constraints come from two 
sources. The first is that there may be fixed target points 
(pins) within the street or avenue. The second is that a 
segment which enters a street, from one side on a specific 
channel must be wired nearer that side than a segment which 
enters the street from the other side on exactly the same 
channel. Constraints of the second kind are called conflicts. 
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Steiner Tree (1=17) Source-Sink (1=23) 



Vertically Biased Chain (1=23) 

Steiner Tree (1=18) 



Minimum A Redundant Graph 

Spanning Tree (1=19) 

Figure 3—Net configurations 


The algorithms used for channel allocation are repeated An example of line packing in a horizontal street is given 

linear assignment of segments to channels using Munkre’s in Figures 12 through 15. 

method 8 and an extension of the simple line pack procedures Net segments entering a street from above in the same 

of Hashimoto and Stevens. The former is used when there channels used by other segments entering the street from 

are pins in the wiring street, the latter when the streets are below are in potential conflict. The net segment entering 
clear. As mentioned earlier, some pin assignments can be from above has to be assigned to a channel above the one 
deferred until the channel allocation phase. assigned to the segment entering from below. This ordering 
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Figure A —Sample global wiring problem 


Figure 8—Global wiring after rerouting 



Figure 5—Initial global wiring 
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Figure 6—Channel demand 
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Figure 10—Vertical segmentation of net N 
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Figure 15—Cyclic conflict breaking 


relation, (i.e., one segment is required to be wired above 
another) constrains the linepacking algorithm and, in some 
cases, leads to a set of constraints which are impossible to 
satisfy at all. Two such situations, the cyclic conflict and 
the chain, are illustrated in Figures 12 and 13. 

It is clearly advantageous to remove these constraints, if 
possible, especially to cut all cycles and over-length chains. 
For this purpose, three straightforward methods are used. 
The first is to interchange logically equivalent pins of a logic 
block (e.g., two inputs of a NAND gate). The second is to 
introduce a short horizontal connection on the vertical plane 
which had the apparent result of moving the entry point of 
a segment into the street. The third is to wire a segment in 
more than one channel. By choosing an uncongested region 
of the street and dividing one horizontal segment into two 
segments with a vertical connector, it is possible to wire the 
two new segments in different channels and thereby satisfy 
previously impossible constraints. An example of such a 
solution is given in Figures 14 and 15. 

The channel allocation program then scans the wire seg¬ 
ments from left to right, filling one channel at a time. When 
one wire segment ends, the next nearest wire segment is 
selected to fill out the channel. This selection is conditioned 
by two considerations: the first is a conflict situation where 
a candidate segment is required to be above or below a 
segment which has not yet been wired. If the wiring of the 
candidate segment in the current channel precludes the wir¬ 
ing of the associated segment in the proper order, then the 
candidate segment is deferred. The second consideration is 
the affinity of the candidate segment to the side of the street 
in which the current channel lies. If two candidate segments 
are equally suitable from the point of view of linepacking, 
the one with the greater proportion of connections to that 
side will take precedence. Figure 16 illustrates the final 
results for the example shown in Figure 12. A “branch and 



Figure 14—Chain breaking 


Figure 16—Line pack solution 
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Figure 17—Histogram of completion percentages 



bound” treatment of the bilateral channel allocation problem 
has appeared in the literature. 5 

RESULTS 

A version of this method was implemented in 1972 to wire 
bipolar chips. The results quoted in Reference 6 will be 
summarized here. 

A representative sample of 100 logic chips each containing 
an average of 125 logic gates and 136 nets. There were 10 
vertical wiring avenues with eight channels each and 10 
horizontal streets with four channels each. The programs 
ran in 325K bytes in an average of eight seconds CPU time 


on a 360/195. The wiring completion percentage in terms of 
segments, was better than 99 percent. The overall utilization 
of wiring space varied between 40 and 55 percent of the total 
length of metal which could be put on the chip. 

The wiring programs exist within the framework of a 
larger physical design automation flow encompassing wira- 
bility analysis, 8 partitioning, placement, 9 interactive wire 
embedding 10 and checking. In almost half the cases, the 
design is fully automatic (Figure 17). In the rest, the inter¬ 
active wire embedding program permits a designer to finish 
the wiring in several hours. 

In the years since 1972, thousands of chips have been 
wired this way. These automatic wiring programs have been 
a key element in the success of the automatic design system 
of which they are a part and the chip technology which they 
support. 
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A speed-oriented, fully-automatic layout program for 
random logic VLSI devices 

by A. FELLER and R. NOTO 

Radio Corporation of America 
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INTRODUCTION 

This paper describes a low cost, quick turnaround capability 
for generating high performance, random logic LSI and 
VLSI devices using the Standard Cell approach. This stand¬ 
ard cell approach, described below, utilizes a fully automatic 
layout capability that automatically maximizes the speed of 
logic paths identified by the user as critical. In spite of the 
sophistication and size of the automatic layout program, the 
system can be run on Midicomputer based systems as well 
as time shared Main Frame Central systems. During the last 
10 years over 1000 custom LSI devices have been fabricated 
using these techniques. 

During this period the basic algorithms used in these au¬ 
tomatic layout programs passed through a series of changes 
due, in no small part, to the needs and demands of a rapidly 
changing LSI technology. Three fundamental changes oc¬ 
curred during this period. 

PLACEMENT, ROUTING, FOLDING PROGRAM 

The first was the development of the Placement, Routing, 
Folding (PRF) program in the 1966-1968 time frame. An 
early PRF layout is shown in Figure 1. The basic approach 
used for placement and routing involved the generation of 
one dimensional geometry. In this algorithm all of the cells 
were placed in a linear row and optimally placed so as to 
minimize crossovers and total wire length. Then the array 
was literally folded by the computer program in a S-type 
function in order to achieve a rectangular aspect ratio. These 
algorithms were developed primarily for two phase dynamic 
PMOS logic. 

SINGLE ENTRY TWO DIMENSIONAL PROGRAM 

(PR2D) 

In the 1969-1971 time frame the second major change in 
the algorithms for the automatic layout programs occurred. 
This was the development of the single entry two-dimen¬ 
sional placement and routing program developed primarily 
for static logic. Its first application, and its principal one, 


was for generating random logic custom LSI devices using 
the metal gate CMOS bulk silicon technology. An illustration 
of a chip designed using this approach is shown in Figure 2. 

With this program the placement function is based on a 
two-dimensional pair interchange of cells on the surface. 
Note, in Figure 2, that except for cells whose input terminals 
face each other, all connections between cells are routed 
around the ends of the cell rows in a wrap-around fashion. 
Because of this routing technique the distance calculation 
for optimum placement based on the two-dimensional pair 
interchange concept is based on an end around calculation 
as opposed to a direct orthogonal calculation. 

As shown in Figure 2, significant chip area is devoted to 
the side channel wiring resulting from the wrap-around in¬ 
terconnections. It is clear that interconnections that passed 
directly through the cell rows, instead of around them, 
would result in a smaller, denser, and faster chip. This 
required far more sophisticated placement and routing al¬ 
gorithms and a more flexible design technique in the layout 
of the standard cell circuitry. Such an approach was devel¬ 
oped in 1973 representing the third major change in the 
evolution of automatic layout programs for LSI and VLSI 
devices. 

DOUBLE-ENTRY TWO DIMENSIONAL PROGRAM 

(MP2D) 

This new approach is called the Multi-port Standard Cell 
Approach. This approach uses circuits called Double-Entry 
standard cells because unlike the previous approaches, con¬ 
nections may be made to these cells at the top or bottom of 
the cell. Figure 3 shows a silicon gate CMOS-SOS Divide 
by Two Counter Cell. This approach is particularly com¬ 
patible with the silicon gate technology. Standard cell fam¬ 
ilies have been developed for silicon gate on bulk silicon as 
well as sapphire substrates. The most significant advantage 
of this approach is that it now permits connections between 
cells anywhere on the chip to be made directly by permitting 
the connections to pass directly through the cell row. This 
is demonstrated in the silicon gate CMOS-SOS LSI device 
shown in Figure 4. Note the significant reduction in the side 
channel wiring with the resulting improvement in chip den- 
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Figure 1—PMOS standard cell array—PRF program 


sity and layout efficiency. Since 1973 this approach has been 
continually improved and is the basic technique that was 
used in the remainder of the LSI and VLSI devices that will 
be described or referred to in the remainder of this paper. 

APPLICATIONS 

Computer oriented applications for which these custom 
LSI capabilities are applicable include ROM decoders and 
sequencers, ALU’s including special purpose ALU’s such as 
an index register modifier, virtual memory control, dynamic 



memory management, polling and error detection and cor¬ 
rection, general high speed random logic, various I/O func¬ 
tions particularly those associated with I/O computers etc. 

SIMPLIFIED CAD SYSTEM 

The block diagram of the CAD system for the Automatic 
Layout and Checking of LSI and VLSI devices which is 
shown in Figure 5 is a simplified version of the actual op¬ 
erating system although it does, for the most part, show the 
principal elements of the system. The only input required 
for the program is a net list containing some means of iden¬ 
tifying the circuit or cell type used and the connectivity. 
Where there is a speed or propagation delay specification, 
that portion of the logic so identified will receive, automat¬ 
ically, special attention in the execution of the automatic 
placement algorithm so as to maximize the speed of those 
designated circuits. 




Figure 2—CMOS standard cell Chip—PRZD program 


Figure 4—CMOS-SOS standard cell array—MPZD program 
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Figure 5—Automatic layout and check system 


After the submission of the input data, the chip will be 
laid out by the Performance Optimized Automatic Layout 
Program. As shown in Figure 5 the user can exercise one of 
two options—the Automatic or the Interactive mode. 

In the Automatic mode the output of the automatic layout 
program, as shown in Figure 5, can proceed to the artwork 
or mask fabrication phase either directly or through a series 
of topology and logic checking programs. In the interactive 
mode, the output of the automatic layout program is edited 
on an Interactive Graphic System (IGS). The designer may 
choose to edit the automatic layout for a variety of reasons 
ranging from area optimization to dynamic performance en¬ 
hancement. The edited layout is then checked for any errors 
in either the topology or the logical connectivity. After val¬ 
idation, the system will automatically generate a magnetic 
tape to generate the final artwork and/or working masks. 

STANDARD CELL LSI APPROACH 

The standard cell LSI approach for generating LSI devices 
begins with the design, layout, and validation of a group of 
custom circuits called standard cells. The divide by 2 
counter cell, whose topology is shown in Figure 3, is a 
typical example of a standard cell. Within the framework of 
the standard cell topology, all the inherent efficiency of a 



Figure 6—“Exclusive OR” standard cell data sheet 
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custom design may be incorporated into these individual 
circuit cells. Once validated, these cells are given an iden¬ 
tification or pattern number and permanently stored. To use 
one of these cells in a logic design, the user calls for these 
cells by pattern number. To aid the logic designer in select¬ 
ing the appropriate cells for optimum logic implementation, 
each cell is completely characterized and documented in 
standard cell data sheets such as that shown in the exclusive 
‘OR’ function in Figure 6. The computer, under automatic 
program control, retrieves the cell data from the stored 
library in a way such that individual cells are represented 
by a collection of polygons corresponding to the various 
mask levels. The automatic layout program then places and 
interconnects the cells in accordance with the prescribed 
logic function. 


DESIGNING LSI DEVICES WITH THE STANDARD 

CELL APPROACH 

Designing LSI chips with the standard cell approach in¬ 
volves only the partitioning of the logic into standard cells, 
the identification of the selected cells and the generation of 
the net list or logical connectivity. This input data is ex¬ 
pressed in a free-formatted, user oriented language called 
the Common Data Base specifically developed such that the 
input data can be programmed by a technician or design 
coordinator. Normally this is the only input data require¬ 
ment. If, however, there are specific logical nets or paths 
whose speed or propagation delay are critical to the proper 
performance of the chip, such paths are identified in the 
input data. The placement algorithm will automatically ori- 



Figure 7—Critical path optimized automatic layout 
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ent the logic cells of the identified critical paths so as to 
minimize the delay associated with the interconnection par- 
asitics. This is described in more detail in the following 
section. 


PERFORMANCE AND SPEED OPTIMIZED 

CAPABILITY 

As indicated earlier, a new capability of the LSI automatic 
layout program is its ability to minimize the propagation 
delays of logic paths identified by the logic designers as 
“critical” logic paths. Figure 7 illustrates this principle. This 
figure shows the computer generated layout of an 8-bit ALU 
with the critical paths highlighted. The closer spacing of the 
cells that form the critical paths results in reduced intercon¬ 
nection parasitics and hence shorter propagation delay. In 
addition to minimizing the delay of the selected ‘critical’ 
paths, the program also computes and generates a detailed 
listing of the parasitic resistances and capacitances associ¬ 
ated with the interconnections. In the very high speed MOS 
technologies, such as CMOS-SOS, these parasitics are a 
determining speed factor. This information can be used to 


TABLE I—Improvement in Automatic Layout Program 


Year 

Size (mils) 

Area 

(mils?) 

Density 

(mils2/device) 

Computer Cost j 

Minutes 

Dollars 

1974 

146x121 

17,666 

18.2 

Not Available 

Not Available 

1975 

129x120 

15,480 

16.0 

13.4 

344 

1976 

115x111 

12,765 

13.2 

5.4 

132 

1977 

100x117 

11,700 

12.1 

Not Available 

Not Available 


update the circuit simulation program or the ‘race’ and ‘tim¬ 
ing’ computations of the Logic Simulation Program. 

DENSITY AND EXECUTION TIME 

In absolute terms substantial improvement has been made 
in increasing the density and reducing the execution time of 
automatically laid out standard cell LSI devices. This im¬ 
provement in the speed and efficiency of the automatic 
placement and routing program is shown in Table I. Table I 
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TOTAL PREDICTED DELAY (IQV)- 112 nsac FROM 
DATA SHEETS BASED ON SIMULATION 




AVERAGE GATE 

DELAY (nste) 

total input to 

OUTPUT (A toBKnstc) 

MAXIMUM COUNTER 

OPERATING FREQUENCY ( MHz ) 

5 V 

10 V 

12 V 

15 V 

5 V 

10 V 

12 V 

15V 

5 V 

10 V 

12 V 

15 V 

MEASURED 

10 

4.9 

4 2 

3 8 

180 

88 

76 

68 

20 

50 



PREDICTED 

14 

6 2 

5 0 

3.9 

250 

112 

90 

70 



60* 

80 


^EXTRAPOLATED 

Figure 9—CAD generated LSI performance 


shows the reduction in area (or increased density) and re¬ 
duced execution time associated with the layout program 
since 1974. As shown in the table the area of the chip was 
reduced by 33 percent, while the execution time was reduced 
by 60 percent. These improvements are due entirely to im¬ 
provements made in the efficiency of the placement and 
routing algorithms. The LSI device used as the test vehicle 
in Table I is a random logic device which is generally the 
type selected for implementation by the standard cell LSI 
automatic layout approach. 

LSI AND VLSI APPLICATIONS 
LSI application 

Figure 8 illustrates a recent LSI application of the stand¬ 
ard cell approach using the CMOS-SOS cell family. This 
chip, which was laid out with the automatic layout pro¬ 
gram, with virtually no manual editing, contains about 550 
gates (2200 transistors) and has a chip size of 235x166 or 
about 17.65 square mils/device. Note that virtually all inter¬ 
connections between cell rows pass directly through the cell 
rows. There are two basic methods which the layout pro¬ 
gram automatically uses in interconnecting cells on non- 
adjacent rows. If in a particular row there is a cell which is 
part of the same node or net as the source gate the program 
will use the poly silicon gate of the cell as a means of crossing 
the cell row. In this case, the polysilicon gate serves a dual 
purpose—that of a control element and a means of crossing 


a cell row. If there is no such cell on a particular cell row, 
the program will automatically move the cells in the appro¬ 
priate area and provide a feedthrough cell to cross the in¬ 
termediate cell row. As shown in the figure considerable 
area is associated with the large tristate drivers and protec¬ 
tive devices in the pad area around the perimeter of the 
chip. The active area of the chip, essentially contained 
within the two parallel bus lines around the inside perimeter 
of the bonding pad area, measures about 202x132 square 
mils or about 12 square mils per device. 

Although this chip is essentially a random logic device, it 
does contain sequential circuits such as ripple counters. 
Figure 9 shows the ripple path through a six stage counter 
in a form in which the path was analyzed by a circuit sim¬ 
ulation program. The table in the lower half of Figure 9 gives 
not only the measured performance of this portion of the 
chip, but also the correlation between measured and com¬ 
puted performance. For example at 10 volt operating level 
the average predicted stage delay, based on conservative 
model parameter values, is 6.2ns while the measured aver¬ 
age delay is 4.9ns. 

VLSI applications (ATMAC microprocessor) 

Two of the VLSI (>1000 gates) devices generated by the 
Standard Cell Automatic Placement and Routing program 
are shown in Figures 10 and 11. In addition to illustrating the 
VLSI capability of the standard cell approach, these chip 
types, which form the basic chip complement for a high 
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Figure 10—VLSI CMOS-SOS microprocessor chip # 1 


performance CMOS-SOS microprocessor, the ATMAC, 
were selected because they both illustrate a new capability 
of the Automatic Layout program called the Subchip Option. 
With the subchip option the program will accept a functional 
design, which may have been remotely designed by conven¬ 
tional manual techniques, (this includes designs on interac¬ 
tive graphic terminals) and automatically make all connec¬ 
tions to it while simultaneously laying out the remainder of 
the host chip. The subchip option recognizes that certain 
logic configurations, such as memories, registers, counters, 
ROMs, and PLA’s characteristically are regular and repeat- 
able structures with a relatively simple interconnection ma¬ 
trix. As such it is not excessively costly and time-consuming 
to achieve a high density layout using conventional layout 


techniques as would be the case for complex, random logic 
devices. 

The net effect of incorporating the subchip option into the 
standard cell automatic layout problem is to combine into 
this computer aided design capability the high density and 
optimization associated with handcrafted design devices 
with the low cost, quick turnaround capability associated 
with automatic layout programs such as the standard cell 
automatic placement and routing capability. 

Each of the VLSI devices shown in Figures 10 and 11 
contains about 5000 transistors, the equivalent of 1250 gates. 
Each chip contains both random and regular logic. The Data 
Execution chip, which performs the arithmetic processing 
and contains the data portion of the microprocessor, in- 
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1 ^' ^ 


Figure 11—VLSI CMOS-SOS microprocessor chip #2 


eludes one subchip function, an 8 word by 8 bit register 
stack. This function contains about 1200 transistors. The 
remainder of the chip, the random logic portion, which con¬ 
tains about 3800 transistors, was fully and completely laid 
out by the automatic layout program. 

The other microprocessor chip, shown in Figure 11, is the 
Instruction and Operand Fetch (IOFU) chip, which func¬ 
tions parallel to the Data Execution chip. It is the control 
element of the system and operates control timing, program 
sequencing, operand accessing and array address updating. 

As shown in Figure 11, this chip contains two subchip 
funcitons. Although one of these subchips is an 8x8 register 
stack, it is a different design from the one in the data exe¬ 
cution chip. The other subchip is a 4 word by 8 bit Last In 


First Out (LIFO) register stack. Together these two hand¬ 
crafted custom designs contain about 2000 transistors. The 
remainder of the IOFU chip, which contains about 3000 
transistors, was automatically laid out and interconnected 
by the automatic layout program which automatically and 
simultaneously made all interconnections to the subchip 
functions as well. 

The block diagram for this two chip 8 bit modular and 
expandible microprocessor is shown in Figure 12. It com¬ 
plements the CMOS/SOS technology with advanced parallel 
architecture to optimize its performance in signal processing 
applications. Its high speed/280 ns and 350 ns instruction 
times is enhanced by many architectural and programmable 
parallelisms that provide an effective throughput of greater 
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Figure 12—CAD generated CMOS-SOS microprocessor block diagram 


than 10 million instructions per second in many array-type 
computations. 

CONCLUSIONS 

Because the standard cell approach for generating random 
logic custom LSI and VLSI devices is based on a completely 
automatic layout capability, it has proven to be a low cost, 
quick turnaround means of generating random logic custom 
LSI and VLSI devices, particularly for the larger and more 
complex devices. 

With its low design cost and quick turnaround capability 
established, emphasis during the last few years has focused 
on improving its high speed performance. Two major im¬ 
provements were made in the program affording higher den¬ 


sity. One was improved placement algorithm that has re¬ 
sulted in a 30 to 40 percent improvement in density. The 
other was the introduction of the subchip option which com¬ 
bines into the standard cell approach the high density and 
custom optimization of handcrafted custom approaches with 
the low cost, quick turnaround capability that characterize 
the automatic layout capability. 
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A methodology for design of digital systems—Supported by 
SARA at the age of one* 
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INTRODUCTION 

There are many practical forces at work which encourage 
improvement in design methods. “Structured program¬ 
ming,” “hierarchical design” and “modularity” are terms 
which have become commonplace. However the frequency 
of their use does not mean that design methods now exist 
which can guarantee that a particular complex system, 
whose programs are shared by many users, will have no 
design errors. Nor do design methods yet exist which can 
guarantee that a digital system, “simple” enough to be in¬ 
tegrated into one device and reproduced in the millions, will 
have no design errors. It is well-known that almost all digital 
systems have so large a set of possible sequences of states 
that great ingenuity must be applied to achieve any mean¬ 
ingful degree of partial verification of their behavior. The¬ 
oretical advances continue to be made in verification. Those 
deriving from concepts of abstract data types 1 are incorpo¬ 
rated in new languages 2-4 and have a lasting influence on 
the way we think about systems. 

Despite those advances it is a source of great embarrass¬ 
ment to computer architects that semiconductor technology 
has created a capability which we cannot now exploit with 
proven design methods. A number of interesting approaches 
to design of concurrent systems are being actively pursued. 
However we do not currently have the languages, operating 
systems and design methods needed to effectively employ 
the new LSI devices which can now be produced. We would 
like to be limited only by economic factors, not technical or 
theoretical factors. 

One hope is that there will evolve effective methods for 
composing such systems from defined, well behaved build¬ 
ing blocks whose composite behavior can be shown to meet 
prestated requirements. A second hope is that starting from 
well formulated requirements and an initial abstract solution 
system, there will evolve helpful guidelines for structural 
partition and effective algorithms for behavioral refinement. 
The refinement procedure should conserve desirable prop¬ 
erties through as many levels of abstraction as the design 
needs. A third hope is that a new dimension for architecture 


* This research was supported by the U.S. Department of Energy, Contract 
No. EY-76-S-03-0034, PA 214. 


of computer systems will emerge from these design methods 
and permit us to effectively use the outpouring of large scale 
integrated devices. 

Realization of these hopes and their joint use are the goal 
of the methodology discussed in the first part of this paper. 
The second part of the paper describes the state of the 
system implemented to support a designer attempting to use 
the methodology. A companion paper 5 illustrates application 
of our methods to design of software. 

The System ARchitects Apprentice, SARA, is only one 
year old. The roots of this work were reviewed in short by 
the author in one of a set of presentatons 6 which announced 
SARA early in 1977. 

We are not so simplistic as to think that we have THE 
only right way to design computer systems. We do feel that 
our methodology makes some significant advance in ap¬ 
proach to design of concurrent software and hardware sys¬ 
tems and has great promise for incorporation of previously 
verified models of software or hardware systems. In fact it 
has long been recognized that methods for design of con¬ 
current systems are also important to good sequential sys¬ 
tem design. The models used to represent desired concur¬ 
rency can also be used to model a set of processes which 
are mutually independent. Sequence can then be imposed 
on them after appropriate analysis rather than by chance. 

As verification methods become more effective and suc¬ 
ceed in avoiding combinational explosion 7 we hope to com¬ 
bine them with the UCLA design methodology and move 
closer to effective use of semiconductor device capability. 


The UCLA design methodology 

We tersely characterize the UCLA design methodology 
as being requirement-driven and supportive of self-docu¬ 
menting design of modular, concurrent, hardware and soft¬ 
ware systems. Figure 1 displays a high level flow chart of 
the design procedure. We will proceed through each step 
discussing what is done with particular attention to implied 
analysis or value judgments made outside of the stated pro¬ 
cedure. In this first part of the paper the reader need make 
no assumption about the nature of automation aids but rather 
should focus on the systematics. 
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Figure 1—UCLA design methodology 
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Initialization of a design 

This design methodology is characterized as requirement 
driven. Given a set of requirements and assumptions about 
the environment, the designer conceives of a system to 
satisfy those requirements. The bulk of our discussion is 
about a procedure which develops sufficient detail about the 
system behavior to determine which of the requirements are 
not satisfied. 

Requirements are not God-given and there continues to 
be extensive study of disciplined analysis techniques 8,9 lead¬ 
ing to well formed requirements. When design costs get too 
high it is often necessary to reconsider the requirements 
themselves. For purposes of this paper we will assume that 
requirements have been carefully generated and that we are 
to make our best effort to design a system which satisfies 
them. It is generally recommended that requirements be 
made at a sufficiently abstract level that they do not pre¬ 
maturely commit to particular implementation methods. 
However, many designers make certain implementation de¬ 
cisions early for practical reasons. For example, a manufac¬ 
turer may restrict implementation to a small prescribed set 
of well tested, well understood building blocks, in order to 
minimize inventory and training costs. These must then be 
carried through the rest of the design process and not be 
compromised. 

Requirements fall into different classes with respect to 
their use in this design procedure. 

Some requirements carry quantitative design constraints, 
such as real time response to a stimulus, which lead directly 
to use of analytical methods or to generation of experimental 
tests. These are the most straightforward requirements to 
deal with, with the exception of two problems: 

• A numerically stated requirement often implies more 
precision than was intended and it is sometimes tedious 
and difficult to introduce tolerances or to manage tol¬ 
erances during evaluation. 

• When a high level system design is partitioned it is 
necessary to determine the set of subsystems which are 
involved in satisfying a quantitative requirement and, 
where possible, to estimate a new quantitative subre¬ 
quirement for each subsystem. Such estimates can turn 
out to be quite arbitrary and to produce undesirable 
distortions of the design. This is a poorly understood 
art. 

Many requirements are qualitative. In some cases their 
satisfaction is determined by delegating the responsibility 
for judgment to individuals who can then record approval 
or disapproval. In other cases qualitative requirements are 
simply passed on to the fabrication phase because they can¬ 
not be evaluated until manufacture. Qualitative require¬ 
ments deserve careful attention because they are often the 
focus of user dissatisfaction. When tradeoff analysis can be 
made to show optimality of a design response to a qualitative 





requirement it can save a great deal of misunderstanding 
and expense. 

An integral part of our design methodology requires initial 
behavioral modeling of the system under design and its as¬ 
sumed environment. The environment model displays only 
those properties necessary to constrain the system design 
and is constructed using the same primitives as those used 
for models of systems being designed. Therefore when two 
independent designs are composed, a designer must test the 
consistency between each subsystem and the others’ envi¬ 
ronment, as informally shown in Figure 2. 

Partition or composition decision 

The decision to compose a model of the system directly 
out of predefined models or to go through another stage of 
top down partition depends upon the availability of prede¬ 
fined building blocks and the designers understanding of 
their behavior. A designer may have a number of building 
block models, which seem to fit a particular design jigsaw, 
but be missing one type of building block. Rather than pass 
through a uniform top-down partition, which always entails 
some overhead, the designer may choose to move aside into 
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an independent design phase. There the designer would es¬ 
tablish requirements for the new building block, successfully 
complete its design and validation using existing building 
blocks, and then return to complete composition along the 
original design path. 

It is of value to add to our intuitive acceptance of what 
building blocks are by the following definitions: 

By building blocks we mean physically realized systems 
with: 

—constrained environments, and 
—predictable input-output behavior, 

such that, given: 

—two or more such compatible systems, and 
—connectivity constraints between them, 

then the combined input-output behavior is predictable. 
Whenever a combined system is also a building block we 
call it well behaved. 

Building block models are representations of building 
blocks if: 

—each model has one or more defined physical realiza¬ 
tions satisfying the definition of a building block, and 
—when building block models are interconnected consis¬ 
tent with interconnection constraints, then combined 
corresponding physical realizations form a physical re¬ 
alization of the combined model. 


Composition 

In the composition step the full set of selected building 
block models in interconnected—to each other and to the 
assumed environment model. Attributes, associated with the 
inputs and outputs of each building block model, provide 
the basis for consistency and completeness checks. These 
checks must include an environment check as noted above 
for Figure 2. The complete model is then analyzed or ex¬ 
periments are conducted in a simulation environment. 

If implementation decisions have not been made earlier 
they are introduced in the composition step building block 
models. 


Composition evaluation 

The evaluation step consists of a combination of analysis 
and experiment. If the behavior of each building block has 
been verified, it may be possible to verify a system require¬ 
ment which has been formally stated. For example, if one 
of the requirements states that a concurrent system must 
satisfy “proper termination” then it is possible that analyt¬ 
ical tools 10 * 11 can be used. More generally the designer will 
generate experiments in a simulation environment and 
choose the most stressful conditions for test. 


Partition 

A partition procedure is used when a set of requirements 
cannot be met directly by composing known building blocks. 
There are different strategies for guiding partition of a sys¬ 
tem into subsystem modules. 

• Pamas 12 has noted that modular decomposition may be 
done according to independent user functions which 
need to be implemented or according to system func¬ 
tions which are needed in carrying out every user func¬ 
tion. The former approach may make mapping of re¬ 
quirements more direct. The latter approach generally 
leads to better utilization of shared resources. 

• One module may be separated from another because 
both design and observation of the included operations 
are considered vital to performance, reliability or cost. 
The module is a system element which is most easily 
observed because its interface with the outside is made 
explicit. 

• A set of modules may be selected because it is estimated 
that they all have about the same complexity, thereby 
providing a design balance. 

• A set of modules may be selected because it is estimated 
that they all have approximately equal influence on 
performance and therefore represent an educated guess 
of a balanced system. 

• Modules may be selected because they are readily com¬ 
posed from known building blocks and therefore will 
avoid another partition step. 

• Modules may be selected because they simplify refine¬ 
ment of higher level behavioral models in a manner 
which guarantees preservation of desired properties. 
Thus, consistency with invariants established in higher 
level proofs of data flow or control flow properties 
would be example criteria. 

Whichever combination of guidelines are practiced leads to 
the creation of two or more structures and their intercon¬ 
nection. For each named subsystem there is then an initial¬ 
ization step identical to that carried out at the previous level 
with the exception that the higher level requirements must 
be mapped onto the new structures. For each named sub¬ 
system a behavioral model is developed which seeks to meet 
the subsystem requirements and hopefully to join in meeting 
full system requirements. 


Partition evaluation 

The set of subsystem models are evaluated by inspection, 
by analysis or by simulation. If any evaluation fails, the 
design is returned to the partition phase for modification or 
re-creation by the designer. If all evaluation steps pass then 
the design is returned to the part of the process where, for 
each subsystem, it is decided that composition can take 
place directly or else that another partition is needed. 
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Implementation 

When the compositions succeed in passing all tests for all 
subsystems, the design moves into an implementation phase 
to prepare the design for fabrication. The following opera¬ 
tions are generally needed: 

• Documentation is prepared. If there have been careful 
records kept during the design process, they provide 
the best description of intended functions, predicted 
behavior, and the usually critical interfaces between 
subsystems. This documentation is important for those 
who will evaluate the fabricated system. However other 
kinds of documentation are generally needed to facili¬ 
tate fabrication, maintenance, marketing, training and 
distribution. Wiring lists, microprograms, bit maps for 
read-only memories, higher level language programs 
and test programs are examples. 

• Changes may have to be made to remove artifact intro¬ 
duced by the modeling and design process. For exam¬ 
ple, instrumentation may have been integrated with a 
model to determine whether performance requirements 
were met but the measurements may themselves dete¬ 
riorate performance and not be deemed necessary. This 
excision must be done very carefully in order not to 
alter desired behavior and sometimes it will force an¬ 
other pass through evaluation of the altered model. 

• The simulation environment may have mechanisms 
which cannot be directly fabricated and are replaced by 
“equivalent” mechanisms. This too must be done very 
carefully lest the modeling and evaluation process be 
invalidated. 


THE SARA (SYSTEM ARCHITECTS APPRENTICE) 

SYSTEM 

The groundwork for SARA was set by Gardner* 13 at the 
beginning of 1975 following several years of collaboration 
between Gardner, Potash and Estrin. The design of SARA 
was begun during the summer of 1975. Implementation of 
structural and behavioral modeling tools continues through 
the present. SARA’s birth was announced at the beginning 
of 1977 6 and a one year old version now marks SARA-rebom 
at MIT-Multics, a more permissive environment than 
UCLA’s IBM/TSO. The infant version of SARA provides 
a collection of modeling tools, at both UCLA and MIT, for 
which some care has been applied to the design of the user 
interface. The sibling SARA’s are reachable by authorized 
users through the ARPANET. 

The current view of the structure of the SARA system is 
shown in Figures 3 and 4. 


* In addition to R. I. Gardner, major contributions to the design and imple¬ 
mentation of SARA were made by D. Berry, I. Campos, C. Carper, J. 
Drobman, B. Fenchel, W. Overman, R. Razouk and W. Ruggiero. 


User interface 

All communication between a designer and the SARA 
DESIGN SYSTEM as well as output to FABRICATION 
are done through 10. 14 The 10 system creates a flexible and 
powerful user-interface and achieves uniformity of interface 
with the various tools in SARA. Moreover it serves as a 
place to contain all machine dependent routines, easing any 
move from one PL/1 environment to another. SELECTOR 14 
is then used to move between language processors at differ¬ 
ent levels as illustrated in Figure 5. 


Structural and behavioral modeling tools 

The most fully tested tools are those contained in the 
modules named STRUCTURES and BEHAVIORS. 

• STRUCTURES holds a language processor called SL1 
which lets a designer interactively describe the struc¬ 
ture of a system using three primitives (see Table I): 
modules, sockets and interconnections. Multilevel 
structural models can be constructed using fully nested 
modules as in Figure 6. Multilevel interconnections, 
can be expressed simply, as displayed in Figure 7. In 
general interconnections may be refined as complete 
systems. 

• BEHAVIORS holds several language processors used 
to create different kinds of models of named structures. 
The principal behavioral modeling tool is the Graph 
Model of Behavior (GMB). The GMB provides a small 
number of primitives (see Table II). The control graph 
primitives allow a designer to interactively create a 
graph explicitly modeling flow-of-control among pro¬ 
cesses including synchronization of concurrent pro¬ 
cesses. The data graph primitives allow a designer to 
display flow-of-data, interpreting the processes in the 
control graph at an abstract level. Furthermore a pre¬ 
processor, called PLIP 15 * 16 allows a designer to con¬ 
cretely interpret each data processor and dataset prim¬ 
itive using the full power of PL1. The topology of 
both the control and data graphs is fixed and they are 
used to model pseudo static aspects of system behavior. 
The PL1 interpretation (PLIP) models dynamic behav¬ 
ior within each processor and causes changes in control 
and data flow along the prescribed data and control 
arcs. Two other tools have been built for analysis of the 
GMB model. An interactive simulation environment 16 
permits experiments to be executed, observed and ana¬ 
lyzed. The simulation environment is used to check 
predictions of results and estimate timing. Moreover, 
the GMB simulator is effective at detecting deadlocks, 
detecting contention for datasets and detecting conten¬ 
tion for processors. One reduction program is used to 
analyze the control flow graph and this analysis can 
determine that a system is “properly terminating”, i.e., 
that it is deadlock free and reentrant. 




Figure 3—Structure of SARA 


TABLE I.—Modeling Primitives 



STRUCTURAL PRIMITIVES 


A NAMED MODULE REPRESENTS AN OBJECT WHOSE INTERNAL, FULLY-NESTED 
STRUCTURE IS HIDDEN FROM THE OUTSIDE. A MODULE'S ONLY POSSIBLE 
COMMUNICATION WITH THE OUTSIDE IS THROUGH A SOCKET. OTHERWISE, 

A MODULE'S NAME IS KNOWN ONLY TO THE STRUCTURE WITHIN WHICH IT IS 
NESTED. 

I 

[EXAMPLE: THE MODULE, CALLED "NAME C", IS COMPOSED OF MODULES 
| "A" AND "B". 


A NAMED SOCKET IS PART OF A MODULE BUT THE SOCKET’S NAME IS KNOWN 
BOTH INSIDE AND OUTSIDE OF ITS HOST MODULE. 

EXAMPLE: THE MODULE "NAME C" IS COMPOSED OF MODULES A WITH SOCKET SA,B 
1 WITH SOCKET SB AND C WITH SOCKET SC. 


A NAMED INTERCONNECTION IS A STRUCTURE WHICH CONNECTS TWO OR MORE 
SOCKETS. 


.EXAMPLE: AN INTERCONNECTION CALLED "LINK" CONNECTS THE SOCKETS 
i A@SA, B@SB, AND C@SC. 


MACHINE PROCESSABLE 


NOTE : SL1 STATEMENT^ M"-Y C E 
MADE IN ANY ORDER. IE” C fl USE 
CREATION OF IMPLIED STRUCTURES. 



NAME_C ; 

A.B.C) 




NAME_C (A@SA, B@S3,C@SC, 


LINK + lAOSA, -@3B. C4SC 
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Figure 4 —Designers view of SARA 


Three other behavioral modeling tools are in various stages 
of integration in SARA. 

A simulation system called DCDS (Digital Control Design 
System) was implemented in 1969 17 and improved in the 
early 1970’s. DCDS is not interactive but allows a designer 
to richly describe a system in terms of logic nets, in terms 
of microprogram sequences or in terms of algorithms. A 
fourth sublanguage called DECLARE is used to combine 
models described separately in the three languages: LOGIC, 


MICROPROGRAM and PL1. DCDS is not integrated into 
the set of interactive SARA tools. Models are translated in 
a batch mode and simulations run in a batch mode. 

The Instruction Set Processor (ISP) modeling language 
created by Bell and Newell 18 has attracted widespread in¬ 
terest. Buliding upon work by Crocker 19 a translator from 
ISP to PLIP was created. One complex machine model was 
built and is still undergoing test. 

The Simulation Oriented Language (SOL) of Knuth and 
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TABLE II.—Behavioral Modeling Primitives 


BEHAVIORAL PRIMITIVES 


MACHINE PROCESSABLE 


A NAMED CONTROL NODE REPRESENTS A STEP IN A PROCESS BEING MODELED. 

A CONTROLLED DATA PROCESSOR (SEE BELOW) MAY BE ASSOCIATED WITH A NODE 
TO PROVIDE INTERPRETATION OF THE PROCESS. 

EXAMPLE : A NODE N1 HAS A SINGLE ENTRY ARC S AND A SINGLE EXIT ARC X. 


A NAMED DIRECTED CONTROL ARC REPRESENTS NON-VOLATILE PREDEDENCE RELATIONS 
BETWEEN SETS OF NODES. IF THERE IS MORE THAN ONE SOURCE NODE OR 
DESTINATION NODE THE ARC IS CALLED COMPLEX; OTHERWISE IT IS CALLED SIMPLE. 
AN ENABLING TOKEN IS PLACED ON AN ARC EITHER AS A STARTING STATE OR UPON 
TERMINATION OF ANY OF ITS SOURCE NODES. WHEN A NODE IS INITIATED, ITS 
ENABLING TOKENS ARE ABSORBED. 

EXAMPLE: A2 AND X ARE SIMPLE CONTROL ARCS. A1 IS A COMPLEX CONTROL ARC 

WHOSE SOURCE SET IS NODES N1 , N2 AND N4 AND WHOSE DESTINATION SET 
IS N5, S, IS AN INCOMING COMPLEX ARC WHOSE DESTINATION SET IS 
Nl, N2 AND N3. 


INPUT CONTROL LOGIC 

A LOGICAL RELATION AMONG THE INPUT ARCS TO A NODE SPECIFIES THE 
PRECEDENCE CONDITIONS THAT MUST BE SATISFIED BY TOKEN STATES FOR THE 
NODE TO BE INITIATED. TOKENS FROM THE INITIATING ARCS WHICH SATISFY 
THE INPUT RELATIONS ARE ABSORBED BY THE TOKEN MACHINE. TOKENS ARE 
ABSORBED FROM ONE OF AN INITIATING ARC SET GOVERNED BY AN OR RELATION 
IN A MANNER ESTABLISHED IN THE TOKEN MACHINE AND FROM ALL MEMBERS OF AN 
INITIATING ARC SET GOVERNED BY AN AND RELATION. 

EXAMPLE: IF ENABLING TOKENS EXIST ON EITHER A1 OR A2 AND ON EITHER A3 
OR A4 THEN Nl CAN BE INITIATED. 

OUTPUT CONTROL LOGIC 

A LOGICAL RELATION AMONG THE OUTPUT ARCS SPECIFIES WHICH ARCS HAVE 
TOKENS PLACED UPON THEM WHEN A CONTROL NODE IS TERMINATED. WHEN AN 
EXCLUSIVE OR OUTPUT RELATION HOLDS, A DATA PROCESSOR INTERPRETATION MUST 
DECIDE WHICH ARC RECEIVES A TOKEN. WHEN AN AND RELATION HOLDS ALL OUTPUT 
ARCS RECEIVE TOKENS. 

EXAMPLE: WHEN Nl TERMINATES, ITS ASSOCIATED CONTROLLED DATA PROCESSOR 

WILL HAVE DECIDED WHETHER TOKENS ARE TO BE PLACED ON B1 AND B2 
OR ON B3 AND B4. 



PCONTROL_GRA D H: 
PNODEF Nl; 
PARCS S,X; 
Nl !":X) ; 
PEND; 


PCONTROL_GRAPH; 

PNODES Nl ,N2,N3,N4,N5; 
PARCS S,A1,A2,X; 

Nl (S:A1); 


N2 (S :A1); 
N3 (S:A2); 
N4 (A2:A1); 
N5 (A1:X); 
SEND; 


INPUT:OUTPUT CONTROL LOGIC 

PCONTROL GRAPHS; 

PNODES Nl; 

PARCS A1,A2,A3,A4,B1 ,B2,B3,B4; 
Nl((A1+A2) * (A3+A4): 

(B1+B2) * (B3+B4)); 

PEND; 


McNeely 20 was implemented in PL1 by the Defense Com¬ 
munication Engineering Center under the name SOL/370. It 
has been installed but not fully tested and integrated. It is 
proposed for use as one of the SARA behavioral model 
analysis tools. 21 




TRANSLATOR PLIP SIMULATOR REDUCER 

Figure 5—SARA—Multilevel view from the selector 


The design library 

Every tool in the SARA system makes use of LIBRARY 
SYSTEM for storing and retrieving models. There are two 
essentially different parts of the LIBRARY SYSTEM. 

• One part contains the public building block library 
which holds models of hardware and software subsys¬ 
tems that have been extensively tested or verified. 
These models have a great deal of value added and are 
assumed to be called upon strongly enough to make the 
investment pay off. Extensive controls on entry and 
modification will be exercised by a library administra¬ 
tor. Drobman has considered the nature of the library 
system with respect to hardware system design and has 
proposed generic building block structures for LSI 
based systems. 

• A second part provides facilities for storage and re¬ 
trieval of private building block models and private 
partial system designs. 

LIBRARY SYSTEM is still in an early stage of de¬ 
velopment. At present we use basic facilities of the 
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TABLE II (Continued) 


I 


_ TYPE _ 

A NAMED CONTROLLED DATA PROCESSOR REPRESENTS A DATA TRANSFORMATION OBJECT 
WHICH IS ACTIVATED WHEN AN ASSOCIATED CONTROL NODE IS INITIATED. E.G., 
PROCESSOR PI IS INITIATED WHENEVER EITHER N1 OR N2 IS INITIATED. WHEN 
PROCESSOR PI TERMINATES IT CAUSES TOKENS TO BE PLACED ON OUTPUT ARCS OF 
THE CONTROL NODE WHICH INITIATED IT. AN INTERPRETATION OF THE DATA 
TRANSFORMATION AND OTHER PARAMETERS SUCH AS TIME DELAY OR RESOURCE 
REQUIREMENTS CAN BE ASSOCIATED WITH THE DATA PROCESSOR. 

EXAMPLE: PROCESSOR PI HAS A RANDOM DELAY ASSOCIATED WITH IT. 


A NAMED UNCONTROLLED DATA PROCESSOR REPRESENTS A DATA TRANSFORMER WHICH 
PROVIDES, AT ITS OUTPUT, STATED FUNCTIONS OF ITS INPUTS INDEPENDENT 
OF CONTROL NODE STATES. IN THE DATA GRAPH AN UNCONTROLLED PROCESSOR IS 
IDENTIFIED BY PROVIDING AN EXPLICIT DECLARATION. AN INTERPRETATION OF 
THE DATA TRANSFORMATIONS AND OTHER PARAMETERS MAY BE ASSOCIATED WITH IT 
IN AN IDENTICAL MANNER TO THE CONTROLLED PROCESSOR. 


A NAMED DATA SET REPRESENTS A PASSIVE COLLECTION OF DATA. DATA 
STRUCTURE MAY BE ASSOCIATED WITH A DATASET. ALL PL/1 DECLARATIONS NOT 
CONTAINING SCOPE OR STORAGE CLASS ATTRIBUTES ARE ACCEPTED AS DEFINITIONS 
OF DATA SETS. CHARACTER STRINGS CANNOT HAVE THE VARYING ATTRIBUTE. 

EXAMPLE: THE DATASET D1 IS A SIX-DECIMAL-DIGIT COMPLEX FLOATING POINT 
NUMBER. 


A NAMED DATA ARC STATICALLY BINDS DATA PROCESSORS AND DATASETS. A DATA 
PROCESSOR HAS READ OR WRITE ACCESS TO A DATA SET IF THE ARROW POINTS TO 
OR FROM THE DATA PROCESSOR RESPECTIVELY. 

EXAMPLE : PROCESSOR PI IS INITIATED BY CONTROL NODE N1. PI READS DATA 
FROM DATASETS D2 AND D3 AND WRITES THEIR SUM INTO DATASET D1. 


©~''' 

0 ^ 


i MACHINE PROCESSABLE 


@OATA GRAPH; 

^PROCESSOR PI (N1 ,N2); 

@END; 

PLIP INTERPRETATION 
OPROCESSOR PI; 

DCL IRAND ENTRY(FIXED BIN(31) 
FIXED BIN(31))RETURNS 
(FIXED BIN(31)); 

/‘RANDOM # GENERATOR*/ 

DCL NUMBER FIXED BIN{31); 
NUM8ER = IRAND(1,2) 

/‘PICK AN INTEGER: 1 OR 
2 */ 

IF NUMBERS THEN @OUTPUT_ARCS = 
1 B1 ,B2 1 ; 

ELSE @OUTPUT_ARCS='B3,B4 ‘ ; 
@DELAY = IRAND (10,100) ; 

/‘PICK RANDOM DELAY FROM 
10 TO 100*/ 

@END PROCESSOR; 




@DATA_GRAPH; 

@UNCONTROLLED_PROCESSORS U1 : 
@END; 


@DATA_GRAPH; 

PDATASETS D1 ; 
@END; 

PLIP INTERPRETATION 


ODATASET D1 COMPLEX 
FLOAT DECIMAL(6); 


@DATA_GRAPH; 

^PROCESSORS P1(N1) ; 
ODATASETS D1 , D2, D3; 
OARCS DAI, DA2, DA3; 
DA3 (D3:P1); 

DA2 (D2 : PI); 

DAI (PI : D1); 

@END; 

PLIP INTERPRETATION 


ODATASET D1 FIXED BIN(31); 
@DATASET D2 FIXED BIN(31); 
ODATASET D3 FIXED BIN(31); 
OPROCESSOR PI ; 

@READ(D3); @READ(D2); 
D1 = D2+D3; 

@WRITE(D1 ); 

@END PROCESSOR; 


interactive operating systems. There has been a great 
deal of design thinking and actual DBMS design but the 
results are too speculative to include here. 


Behavioral—Structural mapping 

A fundamental aspect of the SARA system is the mapping 
of behavior on the structural primitives i.e., modules, sock¬ 
ets, and interconnections. The mapping funcitons are cur¬ 
rently under development. It is only after such mapping that 
SARA tools can detect enought cases of inconsistency and 
incompleteness to repay the designers for the structured 
thinking and excess description that they are forced to pro¬ 
vide when they follow the prescribed methodology. Behav¬ 
ioral attributes are mapped to sockets 5 and focus the de¬ 
signers attention on interface problems, the source of most 
design errors. Then if two sockets are linked via an inter¬ 
connection and an inconsistency in attributes is discovered, 
it is known that either an error was made or else the inter¬ 
action must have a more complex behavior designed into it. 


Multilevel modeling 

As indicated in the first part of this paper the strength of 
support for multilevel modeling procedures and the conse¬ 
quent management of complexity will be the principal test 
of SARA’s reason for existence. During the partition phase, 
the role of SARA is to support evaluation of a refinement 
proposed by the designer and to accept it if the higher level 
abstraction is not contradicted. If the designer insists on the 
refinement, despite contradiction, then a procedure to 
weaken the abstraction is needed. Similarly in the compo¬ 
sition phase, the role of SARA is to compose given building 
blocks into a model and to help discover inconsistencies 
with respect to the next higher level model. 

The behavioral models are clearly more complex. Campos 
and Ruggerio 22 have proposed formal replacement algo¬ 
rithms which retain equivalencies between two graph models 
of behavior. Whenever a single control or data graph prim¬ 
itive is to be refined or whenever a sub-graph can be 
abstracted by a single primitive, the replacement algorithm 
guarantees that changes are localized. In general a refine- 
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SL1 CODE 

S2(S3,S1); S2(S3@L2,S1@L2) ; S2 .L2+(S3@L2 ,S10L2); 
S3(S4,S40L2); S3.L2(S4@L2 ,S3@L2); 

SI(SUB1,SUB2,SUB3 ,SUB1@L2 ,SUB1@L1,SUB2@L1,SUB3@L1); 

SI.L1+(SUB1@L1,SUB2@L1,SUB3@L1); 

S1.L2+(SUB1@L2,S1@L2); 

NOTE : THE IS USED TO DESCRIBE NESTING IN A FULLY 
QUALIFIED NAME, E.G. S2.S1.SUB1 IS THE FULLY QUALIFIED 
NAME OF SUB1 IN FIGURE 6. 



S2 


Figure 6—Multi-level modules 


ment of (or abstraction into) a data graph primitive implies 
modification of the associated control graph—and vice 
versa. Furthermore it is possible to retain “well-behaved” 
properties like proper termination during these replace¬ 
ments. These replacement algorithms are restricted in the 
sense that we do not have an algorithm yet to permit re¬ 
placement of an arbitrary sub-graph by another sub-graph. 
However they add considerable strength to SARA in support 
of the designer. 

In the same vein, our models of concurrent systems pro¬ 
vide a context in which to embed verified processor inter¬ 
pretations and seek algorithms to prove partial correctness 
of a full data graph including interpretation. 


For effective multilevel modeling, the “bottom line” is 
the accuracy, clarity and richness of the building block 
models available to the designer. No matter how late or 
early implementation decisions are explicitly made, the de¬ 
sign process must be goal oriented and the designer must 
either understand the elements or have faith in the models 
reinforced by successful use of them. For hardware building 
block models, we find ourselves giving a great deal of atten¬ 
tion to detail in order to convince ourselves that our models 
can accurately represent manufacturers specifications. 23 
Drobman 24 has been able to generalize such study by defin¬ 
ing “generic” as well as specific building blocks. There are 
other ways to harness rich populations of building blocks. 
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SL1 CODE - REFINED MODEL 

SUB10L1 (SUB10L1 .WIRE1 ,SUB1@L1 .MIRE2,SUB1 (PL1 .WIRE3); 
SVB20L1 (SUB2@L1.WIRE3,SUB2@U .WIRE2); 

SUB30L1 (SUB3@L1.WIRE2,SUB3@L1.WIRE1); 

LI.WIRE! + (SUB10L1.WIRE1,SUB3@L1.WIRE); 

L1.WIRE2 + (SUB1@L1.WIRE2,SUB2.@Ll.WIRE2,SUB3.@L1.WIRE2); 
L1.WIRE3 + (SUB10L1.WIRE3,SUB20L1.WIRE3); 




Figure 7—Multi-level interconnection 


SL1 CODE - HIGH LEVEL MODEL 

UNIVERSE (SUB1,SUB2,SUB3); 

UNIVERSE (SUB10L1,SUB2@L1 ,SUB3@L1); 
LI + (SUB1@L1,SUB2@L1,SUB3@L1); 


For example we have a path for translating ISP descriptions 
into PLIP interpretations but we do not yet fully understand 
limitations imposed by efficiency issues. 

In the companion paper, 5 a software example is taken all 
the way from programming-in-the-large to programming-in- 
the-small or PL1 code. In that case the PL1 language defi¬ 
nition provides the building blocks and the transition is 
smooth because the entire SARA System is written in PL1. 
When concurrency exists, the transition is not completely 
smooth because the “token machine” is an integral part of 
the SARA environment and there must be equivalent ma¬ 
chinery wherever a SARA model is expected to execute. In 
fact we are exploring new architectures in which SARA 
models would run with no alteration of the designed and 
tested model. 


SUMMARY AND CONCLUSION 

The UCLA design methodology has been elaborated in an 
attempt to indicate its requirement-driven and self-docu¬ 
menting properties for modular, concurrent, hardware and 
software systems. The need for implementation in hardware, 
for example, may show up explicitly in initialization or fi¬ 
nally appear in composition. 

Those goal characteristics which distinguish this design 
philosophy are: 

• Explicit modeling of environments using the same prim¬ 
itives as the systems being designed. 

• Separate but related static and dynamic models. 

• Separate but related control and data flow models. 
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• Separate but related structural and behavioral models. 

• Support for both top-down and bottom-up design. 

• Implementation decisions forced in composition but re¬ 
tained when they are entered earlier. 

• Supported “well-behaved” properties in multilevel 
modeling and analysis. 

• Supported simulation of concurrent processes. 

• Supported fabrication. 

SARA is useful now as a collection of tools which help 
structured design of concurrent systems. SARA has been 
tested in a number of design problems. As a result of its 
recent move to MIT-MULTICS, it is soon to be tested on 
larger system problems. To the extent that support of the 
requirement-driven multilevel design methodology is in¬ 
creased and detected “errors” are reported to the designer, 
SARA’s value will increase significantly. A number of prob¬ 
lems with the current methodology and its support are dis¬ 
cussed in the companion paper. 5 

We believe that SARA is unique in the support it gives to 
design of concurrent systems. We do not look on this meth¬ 
odology as being in competition with other active work in, 
for example, dataflow architecture or sequential software 
design. We are constantly alert to incorporation of other 
maturing synthesis techniques, particularly when any of our 
designs reach the stage of incorporating sequential pro¬ 
grams. 

The number of poorly understood problems which have 
been pointed to in this paper is very large. They give cre¬ 
dence to the author’s philosophy which rejects completely- 
automatic design at this time and seeks to support the de¬ 
signers injection of value judgments and decisions. Auto¬ 
mated design procedures should be introduced selectively 
and patiently. Sarah begat Isaac when she was ninety years 
old [Genesis]. It is possible that, with the help of SARA and 
the help and the wisdom of many researchers and machines, 
we will know enough before 2067 to beget a powerful ISAAC 
(Information System Automatic Architect Computer). We 
do know how to beget an ISAAC but also know that a 1978 
ISAAC would be fired for incompetence long before tenure 
could be achieved. 
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SARA aided design of software 
for concurrent systems* 


by IVAN M. CAMPOS** and GERALD ESTRIN 
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Los Angeles, California 

INTRODUCTION 

In the past several years it has often been noted that soft¬ 
ware is the major cost component in designing, implement¬ 
ing and maintaining computer systems. Unreliability and 
associated debugging and maintenance activities are among 
the main elements responsible for this increase in cost. Most 
bugs are “born” in the design phase, i.e., the design itself 
is usually faulty, 1 and when a bug is uncovered some 
changes are likely to occur. Whenever modification is made, 
it is generally difficult to keep track of all consequences that 
it may provoke. Poor design often shows up as a high degree 
of connectivity among subsystems. 2,3 Interconnections may 
be “hidden,” representing unstated assumptions that a sub¬ 
system makes about other subsystems and leading to un¬ 
foreseen side effects when seemingly harmless changes are 
made. 4,5 

In an attempt to improve software, extra effort tends to 
be invested in design. Considerable attention is being given 
to issues such as stepwise refinement, 6,7 structured program¬ 
ming, 8 modularity, 9,10 and composite design. 3 Data abstrac¬ 
tion concepts have been introduced into languages such as 
CLU, 11 EUCLID, 12 and ALPHARD. 13 Active research con¬ 
tinues to uncover more effective methods for analysis and 
synthesis of verified programs. 14 

Since the early 1960’s, Estrin and his colleagues have 
been involved in the development of models of computa¬ 
tional algorithms, models of interpreters, and procedures for 
mapping one on the other. The UCLA Graph Model of 
Computation; 15-21 the development of a methodology 
(DCDS) for design of hierarchical hardware and firmware 
systems; 22 and the revelation of difficulties in evaluation of 
unstructured computer systems 23 led this UCLA group to 
concentrate on synthesis of structured systems, in which 
one might manage complexity from the outset and prepare 
for further analysis. 

The major direction of work at UCLA on methodology of 
synthesis is to complete a set of tools which support a 
structured multilevel design procedure for software or hard¬ 
ware development. 24 This interactive computer-aided sys- 


* This research was supported by the U.S. Department of Energy, Cont 
No. EY-76-S-03-0034, PA214. 

** Presently at: Ivan Moura Campos, Departamento De Ciencia Da Com- 
putacao, Universidade Federal De Minas Gerais, Pampulha, Belo Hori¬ 
zonte—MG, 30 000 Brasil 


tern called SARA (Systems ARchitect’s Apprentice) pro¬ 
vides languages to help a designer form useful abstractions 
which can be manipulated and tested in a disciplined 
way, 25-28 and which may be forced to retain good proper¬ 
ties. 29 

SARA supports both a bottom-up composition (abstrac¬ 
tion) procedure and a top-down partitioning (refinement) 
procedure. In both cases, however, the procedures are re¬ 
quirement-driven so that attention can be focused on the 
gap between a designer’s declared intent and the current 
approximation of the system’s behavior. 

Models are created using a small number of primitives* 
whose manipulation are supported by SARA interactive 
translators and a simulator (see Tables I and II in the com¬ 
panion paper 24 ). 

Modules, sockets and interconnections are structural 
modeling primitives which allow creation of a hierarchical 
name space for a fully nested system. Named modules per¬ 
mit interaction only through sockets. Socket names are 
known both inside and outside of modules, thus supporting 
declarations of interfaces. Named interconnections support 
interaction among modules. 

Explicit control flow behavior can be expressed by control 
nodes, which span initiation and termination of associated 
processes, and by directed control arcs, which are used to 
express precedence conditions. Control conditions and 
thresholds can be prescribed to condition process initiation. 
Control consequences can be prescribed to take effect fol¬ 
lowing process completion. Tokens mark the state of control 
flow and a token machine governs progress of the state of 
a model. Data flow is described by graphs composed of 
controlled processors, uncontrolled processors, datasets and 
data arcs. 

All of the above primitives are currently used to form 
models which are static in the sense that the topology cur¬ 
rently does not change during execution. Interpretations can 
be associated with the data flow primitives (processors and 
datasets) and provide the semantics for modeling of dynam¬ 
ics. Currently, a pre-processed PL/I (PLIP) is the interpre¬ 
tation language. An interactive simulator supports experi- 


* The authors chose not to provide, in this paper, the full detail of syntax 
and semantics implemented in SARA. The interested reader can find such 
information in Refemce 26. We have tried to include enough detail to follow 
examples. Note that the “@” symbol is used for many purposes: to identify 
a socket name or a declaration or language processor. 
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ments on models. Extensive discussion of SARA modeling 
may be found in the references noted above. 

The objective of this paper is to expose the SARA meth¬ 
odology as applied to software. The central problem is the 
set of tools and procedures to be brought to bear on building 
possibly concurrent software systems which reliably carry 
out the designer’s intent. 

We also wish to propose a role of SARA models not only 
as design and evaluation tools, but also as the basis for a 
superstructure to be overlaid on the actual code of a de¬ 
signed product as redundant information for the operating 
system. One consequence of imposing such a superstructure 
on source code modules is that it provides its operating 
system and its user with a “live” documentation whose 
objective is to enforce consistency between requirements, 
structure, function and behavior of the software system. 

In the body of the paper below, we take the reader through 
the key steps of two example designs and try to generalize 
the procedure through commentary. We chose the design of 
a synchronized sender-receiver process and an abstracted 
FFT. They are relatively simple examples which are of some 
importance and serve to bring out many properties of our 
design method and the SARA system. 

Following an informal statement of need, the design uni¬ 
verse is partitioned into the system to be designed and its 
environment. An initial behavioral model (of the system and 
of the environment) fixes all assumptions under which the 
design is to proceed. Requirements establish evaluation cri¬ 
teria for the system. Following “top-down” levels of de¬ 
signer-created behavior, there is a final composition step to 
produce PL/I code. In the models below, comments will 
highlight SARA conventions. In all other cases PL/I con¬ 
ventions will be assumed to hold. 


A DESIGN EXAMPLE—A SIMPLE MESSAGE 
TRANSMITTER 

The informal statement of our design problem is “design 
a system whereby a process produces and sends a sequence 
of data to another process, if the latter is not busy. 


DEFINING THE STRUCTURE 

The initial design steps in the SARA design procedure 24,27 
lead us to partition our UNIVERSE into two SL1 modules, 
MESSAGE TRANSMITTER, the system to be designed, 
and ENV, its environment. The machine processable SL1 
code, its graphical representation and the requirements for 
both ENV and MESSAGE TRANSMITTER are shown in 
Figure 1. Parentheses establish nesting level so that the 
second line of SL1 code is read “ENV” is composed of 
three sockets, called @ INFILE, @ HERALD, and 
@OUTFILE.” Identifiers starting with the character ‘@’ 
are socket names. The symbol ‘+’ indicates that the system 
to its left is an interconnection which connects the sockets 
whose names appear between the parentheses that follow it. 
Thus the third SL1 statement says that “within UNIVERSE 
there is an interconnection named SL1 which connects the 
socket ENV @INFILE to MESSAGE TRANSMITTER 
@INFILE.” 

The three interfaces between the modules that constitute 
ENV and MESSAGE TRANSMITTER are called HER¬ 
ALD, INFILE and OUTFILE, respectively, as shown in 
the sockets. 


SL1 CODE FOR STRUCTURAL MODEL OF UNIVERSE 


UNI VERSE(ENV.MESSAGE_TRANSMITTER); 

ENV(@INFILE,@HERALD,@OUTFILE);MESSAGE_TRANSMITTER(@INFILE,@HERALD,@OUTFILE) ; 
UNIVERSE!LI+(ENV@INFILE,MESSAGE_TRANSMITTER@INFILE)); 
UNIVERSE(L2+(ENV@HERALD,MESSAGE_TRANSMITTER@HERALD)); 
UNIVERSE(L3+(ENV@0UTFILE,M£5SAGE_TRANSMITTER@0UTFILE)); 


RE1 - ENV CONTAINS TWO FILES. ONE READ-ONLY 

FILE NAMED INFILE AND ONE WRITE-ONLY FILE 
NAMED OUTFILE. 

RE2 - ENV PROVIDES AN INTERPRETER FOR HERALD 
THAT PASSES CONTROL TO IT AND RECEIVES 
CONTROL FROM IT UPON COMPLETION. 

RE3 - BOTH INFILE AND OUTFILE ARE ORGANIZED 
AS RECORD, SEQUENTIAL, ENVIRONMENT 
(U(800) BUFFERS(1) CONSECUTIVE). 

RE4 - EACH RECORD OF BOTH INFILE AND OUTFILE 
HOLDS A CHARACTER STRING 255 CHARACTERS 
LONG. 

RE5 - THE LAST RECORD TO BE READ FROM INFILE 
STARTS WITH THE CHARACTER STRING 


UNIVERSE 


MESSAGE TRANSMITTER 

MESSAGE_TRANSMITTER ENCOMPASSES TWO PROCESSES, A 
SENDER AND A RECEIVER. 

SENDER AND RECEIVER ARE SYNCHRONIZED, I.E., SENDER 
ONLY SENDS A NEW MESSAGE WHEN RECEIVER IS IDLE AGAIN. 

MESSAGES TO BE TRANSMITTED ARE 255 CHARACTERS LONG. 
THE LAST MESSAGE STARTS WITH THE STRING 



0INFILE 


OINFILE 


OrU 


RMl - 

RM2 - 

RM3 - 


Figure 1—Requirements superimposed on ENV and message-transmitter 
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THE BEHAVIORAL MODEL GMB (GRAPH MODEL 
OF BEHAVIOR) 

Figure 2 contains the graphical representation of the GMB 
model of the interaction between ENV and MESSAGE 
TRANSMITTER, its machine processable representation 
(upper left), the PLIP interpretation associated with the da¬ 


tasets and the processor of the data graph (upper right), the 
explicit mapping of GMB primitives into the structure (lower 
left) and the list of attributes for each socket (lower right). 
Notice that the correspondence between node N2 and pro¬ 
cessor P2 is indicated in the source GMB code in both the 
“©nodes” statement and the “©processor” statement. The 
initial state of the graph is shown to consist of a single token 


DESCRIPTION OF life INTERPRETED 6MB 


INTERPRETATION OF DATA GRAPH 


@C0NTR0L_GRAPH 

@N0DES N1, N2(°2), N3; 
@ARCS S(l), At, A2, X; 
N1(S:A1); 

N2(AT:A2); 

N3(A2: X); 
i?END. 

0DATAGRAPH; 

OPROCESSORS P2(N2); 
(BDATASETS INFILE, OUTFILE; 
@ARCS DAI, DA2; 

DAT(INFILE:P2); 

DA2(P2:OUTFILE); 

@END; 


GMB TO SL1 MAPPING 


GMB PRIMITIVE 

SLI PRIMITIVE 


NAME 

TYPE 

NAME 

TYPE 

S 

C ARC 

ENV 

MODULE 

NI 

C NODE 

ENV 

MODULE 

A1 

C ARC 

ENVQHERALO 

SOCKET 

A2 

C ARC 

ENV@HERALD 

SOCKET 

N3 

C NODE 

ENV 

MODULE 

X 

C AfiC 

ENV 

MODULE 

INFILE 

DATASET 

ENV 

MODULE 

DAI 

D ARC 

ENV@INFILE 

SOCKET 

DA2 

D ARC 

ENV@0UTFILE 

SOCKET 

OUTFILE 

DATASET 

ENV 

MODULE 

A1 

C ARC 

L2 

INTERC 

A2 

C ARC 

L2 

INTERC 

DAI 

D ARC 

LI 

INTERC 

DA2 

D_ARC 

L3 

INTERC 

A1 

C ARC 

MESSAGE TRANSMITTER@HERALD 

SOCKET 

A2 

C ARC 

MESSAGE TRANSMITTER@HERALD 

SOCKET 

N2 

' C NODE 

MESSAGE TRANSMITTER 

MODULE 

DAI 

D ARC 

MESSAGE TRNASMITTEP.QINFILE 

SOCKET 

P2 

PROCESSOR 

MESSAGE TRANSMITTER 

MODULE 

DA2 

D ARC 

MESSAGE TRANSMUTER@0UTFILE 

SOCKET 


@DATASET LNFILE FILE RECORD SEQUENTIAL INPUT 
ENVIRONMENTS (800) BUFFERS (I) CONSECUTIVE); 
QDATASET OUTFILE FILE RECORD SEQUENTIAL OUTPUT 
ENVIRONMENT (U(800) BUFFERS(I) CONSECUTIVE); 
^PROCESSOR P2; 

DCL MESSAGE CHAR (255); 

OPEN FILE(INFILE); 

OPEN FILE(OUTFILE); 

/* SENDER PROCESS */ 

LQPP : READ FTLE(INFILE) INT0(MESSAGE); 
@RLAD(INFILE);. 

/* RECEIVER PROCESS */ 

WRITE FILE(OUTFILE) FROM(MESSAGE); 

@WRITE(OUTFILE); 

/* TEST FOR COMPLETION */ 

IF SUBSTR(MESSAGE,1,2)-i= '**' 

THEN GO TO LOOP; 

CLOSE FILE(INFILE); 

CLOSE FILE(OUTFILE); 

@ENDPR0CESS0R; 


SOCKET ATTRIBUTES 


SLI SOCKET 

ATTRIBUTES 

ENVOINFILE 

DATA OUTPUT 


CHARACTER (255) 

NONVARYING UNALIGNED 

STATIC EXTERNAL 

ENVOHERALD 

CONTROL INPUT-OUTPUT 

ENV@OUTF T LE 

DATA INPUT 

CHARACTER (255) 

NONVARYING UNALIGNED 

STATIC EXTERNAL 

MESSAGE TRANSMITTER0INFILE 

DATA INPUT 


CHARACTER (255) 

NONVARYING UNALIGNED 
STATIC EXTERNAL 

MESSAGE TRANSMITTER@HERALD 

CONTROL INPUT-OUTPUT 

MESSAGE TRANSMITTEROOUTFILE 

DATA OUTPUT 


CHARACTER (255) 

NONVARYING UNALIGNED 
STATIC EXTERNAL 




”^^^DA2 



’’ESSAGE-TRANS’IITTER 


Figure 2—High level model of message—transmitter and its environment 
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on arc S. The expression in parentheses after node names 
contain arc expressions separated by a colon. To the left of 
the colon one specifies the input logic expression to the 
node (precondition for its activation); conversely to the right 
of the colon one specifies the output logic expression 
(postcondition to its deactivation). Any well-formed logic 
expression can be used. If the designer is restricted to using 
the operators AND (“*”) and OR (“ + ”) then analysis of 
control graph properties is made easier. Let us now consider 
evaluation of the model with respect to the requirements. 
RE1 and RE3 are satisfied by the first two PLIP dataset 
interpretations. RE2 is satisfied by the control graph de¬ 
scription. RE4 is satisfied by the first DCL in the process 
P2 interpretation. RE5 is implied by the interpretation of 
processor P2. RM1 is satisfied by inspection. RM2 is satis¬ 
fied because the Sender and Receiver processes are done 
sequentially. RM3 can be tested in the SARA simulator 
environment and, finally, on a PL 1 machine when the pro¬ 
gram is composed. 

The above model is read by the SARA translator and 
simulated, even at this high level, by providing test messages 
in INFILE and observing at the end of simulation that they 
appear in OUTFILE in a manner that satisfies the require¬ 
ments. 


CODE SYNTHESIS—BOTTOM-UP COMPOSITION 


This design example is so simple that we directly go bot¬ 
tom-up in the design process. 

A few observations are in order. The fact that PLIP is an 
extension to PL/I via a preprocessor places us in a privileged 
position to directly synthesize PL/I code. We will try to 
treat each PLIP statement as a building block model of the 
corresponding PL/I statement. We assume that we have 
validated MESSAGE TRANSMITTER with respect to re¬ 
quirements and are therefore ready to directly synthesize 


INTERPRETATION OF DATA GRAPH 


PLIP 

0DATASET INFILE FILE RECORD SEQUENTIAL INPUT 
ENVIRONMENT(U(800) BUFFERS(1) CONSECUTIVE); 
ODATASET OUTFILE FILE RECORD SEQUENTIAL OUTPUT 
ENVIRONMENT (U(800) BUFFERS(l) CONSECUTIVE); 
QPROCESSOR P2; 

DCL MESSAGE CHAR (255); 

OPEN FILE(INFILE); 

OPEN FILE(OUTFILE); 

/* SENDER PROCESS */ 

LOOP; READ FILE(INFILE) INTO(MESSAGE); 
@READ(INFILE); 

/* RECEIVER PROCESS */ 

WRITE FILE(OUTFILE) FROM(MESSAGE); 
OWRITE(OUTFILE); 

/* TEST FOR COMPLETION */ 

IF SUBSTR(MESSAGE,1 ,2)i= '**' 

THEN GO TO LOOP; 

CLOSE FILE(INFILE); 

CLOSE FILE(OUTFILE); 

0ENDPR0CESS0R; 


the code. This can be seen as a “fabrication phase” after 
the modeling has been completed, and we assume that future 
development of SARA might make this process automatic. 

We compose the MESSAGE TRANSMITTER procedure 
by declaring as internal variables all those datasets that have 
been mapped separately on ENV in the GMB-to-SLl map¬ 
ping (in this case INFILE and OUTFILE). We concatenate 
them with the declarations of the internal variables found in 
the PLIP code of P2. 

The next “cleaning-up” step consists of deleting all the 
@READ and @WRITE statements from the code, since 
they are only used by the SARA GMB simulator. The final 
code is shown in Figure 3 next to the original PLIP code 
(taken from the upper right box in Figure 2). 

This was, of course, a trivial example but served to illus¬ 
trate the methodology. Our second design example is more 
complex. 

A SECOND DESIGN EXAMPLE—A FAST FOURIER 

TRANSFORMER 30 

In this example we wish to focus the readers attention on 
the composition procedure, which is our name for a bottom- 
up design procedure. 

A signal processing problem which has received extensive 
attention can be informally stated as “design a system which 
receives sequences of complex numbers; calculates the Fast 
Fourier Transform of each of those sequences; and makes 
it available to the environment that issued the original com¬ 
plex numbers.” For purposes of this paper we consider a 
reduced problem for sequences of 8 complex numbers. It is 
of some importance that the reader extrapolate the appli¬ 
cation of the method to larger problems. 

In this design example we get a little more formal in our 
discussion. We use a specification of the problem (which we 
call our invariant) whereby a description of the desired in- 


PL1 CODE _ 

MESSAGE TRANSMITTER: PROCEDURE 

DCL TNFILE FILE RECORD SEQUENTIAL INPUT 
ENVIRONMENT^ (800) BUFFERS(1) CONSECUTIVE); 
DCL OUTFILE FILE RECORD SEQUENTIAL OUTPUT 
ENVIRONMENT(U(800) BUFFERS(1) CONSECUTIVE); 
DCL MESSAGE CHAR (255); 

OPEN FILE(INFILE); 

OPEN FILE(OUTFILE); 

/* SENDER PROCESS */ 

LOOP: READ FILE(INFILE) INT0(MESSAGE); 

/* RECEIVER PROCESS */ 

WRITE FILE(OUTFILE) FR0M(MESSAGE); 

/* TEST FOR COMPLETION */ 

IF SUBSTR(MESSAGE1 ,2)~>- '**' 

THEN GO TO LOOP; 

CLOSE FILE(INFILE); 

CLOSE FILE(OUTFILE); 

END MESSAGE TRANSMITTER 


Figure 3—Original PLIP interpretation and final composed PL1 code for 
message-transmitter 
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teraction between the system to be designed (FFT) and its 
environment (ENV) is produced. This invariant is composed 
of: 

• A structural model which splits the design universe into 
two subsystems, ENV and FFT, and shows their struc¬ 
tured connections by means of interconnected sockets. 

• A list of requirements for both the system under design 
and its environment. 

• A graph model of behavior (GMB) of the desired inter¬ 
action between ENV and FFT. 

• An interpretation associated with the GMB, which 
shows in procedural form which input/output transfor¬ 
mations are to be performed by both ENV and FFT, 
with particular attention to interface issues. 

THE STRUCTURAL MODEL 

The machine processable and graphic SL1 model of our 
design universe is presented in Figure 4 where we distinguish 
two subsystems of UNIVERSE, namely ENV and FFT. 


Structurally, ENV and FFT are connected through inter¬ 
connections XI and FI, each of which interconnect a socket 
in ENV to a socket in FFT. These interconnections support 
both control and data interactions between the two subsys¬ 
tems. In Figure 4 we have chosen to show both a top level 
structural model and a more detailed socket model. We are 
anticipating modeling concurrent programs. Eight of the in¬ 
terconnections support flow from ENV to FFT, and the 
other 8 support flow from FFT to ENV. 

REQUIREMENTS 

The requirements are superimposed on the structural 
model in Figure 4. They are separated into two sets, one 
mapped onto ENV and the second onto FFT. 

THE BEHAVIORAL MODEL 

The flow-of-control and the flow-of-data graphs are shown 
in Figures 5 and 6 respectively, mapped onto the ENV and 


TOP LEVEL SL1 


REFINED SL1 


UNIVERSE (ENV,FFT); 

ENV (9XI,9FI); 

FFT (9X1 ,9FI); 

UNIVERSE (XI+(ENV@XI,FFT9XI)); 
UNIVERSE (FI+(ENV9FI,FFT9FI)); 


UNIVERSE (ENV,FFT); 

ENV (9X0,9X1,9X2,9X3,9X4,9X5,9X6,9X7,9F0,9F1,9F2,9F3,9F4,9F5,9F6,9F7); 

FFT (9X0,9X1,9X2,9X3,9X4,9X5,9X6,9X7,9F0,9F1,9f2,9F3,9F4,9F5,9F6,9F7); 

UNIVERSE (X0+(ENV9X0,FFT9X0), X1 + (ENV9XI.FFT9X1), X2+(ENV9X2,FFT9X2), X3+(ENV9X3,FFT9X3), 
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Figure 4 —Structural model and requirements for FFT system showing SL1 
for both abstract and refined socket levels 
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UNIVERSE 

Figure 5(a)—High level FFT control graph 



Figure 6(a)—High level FFT data graph 


FFT structures. Figures 5A and 6A show the top level 
models. Figures 5B and 6B show more refined models. The 
machine processable representation of the GMB, the GMB 
to SL1 mapping, the PLIP interpretation and the socket 
attributes are generated in the same manner as for the design 
example MESSAGE TRANSMITTER. The detail is not in¬ 
cluded in this paper but is available on line at MIT-Multics. 
These high level models can be reduced to PL1 code in the 
same manner as the earlier example. 

Let us assume that our tests check that the simulation 
model is proper and we consider composition utilizing a 
“butterfly” building block. 

THE BUTTERFLY BUILDING BLOCK 

The following description should give the reader a firmer 
idea of what is meant by a software building block in SARA. 
We continue our discussion with some informality in order 
to conserve space and to be readable. A building block 
model includes a name-set, an attribute set, structural 
models and behavioral models. 

The name set provides a place to list all alias library 
names. In this case we call our building block FFT BUT. 
However the library might also respond to aliases like FFT 
BUTTERFLY or FFT BMOD. 



[universe 


The attribute set includes an informal description and 
comments about interpreters and versions. For this case the 
attribute set reads: 

Informal description 

The FFT “butterfly” whose non-procedural model and 
code are shown below in PL/I-like syntax is a software 
module designed to run in a multiprocessing environment 
(concurrently with other such computations) and perform 
one complex multiplication, one complex addition and one 
complex subtraction. The latter two operations are done in 
parallel within the butterfly. 

Its usefulness is based on the existence of highly parallel 
algorithms that can be composed by binding several such 
procedures together in ingenious ways in order to compute 
the Fast Fourier Transform of arrays of complex numbers. 
The input and output assertions for the Butterfly are 

10**(—70)4=abs(X0)<= 10**70 
10**(—70)<=abs(Y0)<= 10**70 

X1=X0+WPY0, where WPY0 is a complex constant 
Y1=X0-WPY0, where WPY0 is a complex constant 
10**-70<rabs(WPY0)<= 10**70 



Figure 5(b)—Refined control graph 


Figure 6(b)—Refined FFT data graph 
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FFT BUT: PROCEDURE; 

DCL (XO, YO, XI, Yl) COMPLEX DECIMAL 
FLOAT (6) 

EXTERNAL; 

DCL (A3, A4, A7, A8) SEMAPHORE; 

DCL WP COMPLEX DECIMAL FLOAT(6) INITIALS); 
DCL WPYO COMPLEX DECIMAL FLOAT(6); 


WAIT(A3 & A4) 
WPY0=WP * YO; 
COBEGIN 
X1=X0+WPY0; 
Y1=X0—WPYO; 
COEND; 

SIGNAL (A7 & A8); 
END FFT BUT; 


Interpreters and versions 

This procedure runs in PL/I multitasking environments, 
provided the availability of the following Concurrent PAS¬ 


CAL 31 * 32 constructs: 

cobegin . . . coend 

signal (semaphore expression) 

wait (semaphore expression) 

An ingenious implementation of Brinch Hansen’s sema¬ 
phores as an add-on to the PL/I machine is given in Refer¬ 
ence 33. 

We notice here that the construct cobegin . . . coend can 
trivially be implemented with semaphores. For each con¬ 
currently initiated task we issue a “signal(semaphore)”, 
having a unique semaphore per task. Inside of each of those 
tasks, the first executable statement is a “wait(semaphore)” 
which waits for the appropriate semaphore. Upon comple¬ 
tion, each task issues a “signal(completion semaphore)”, 
one semaphore per task. The “coend” construct is then 
substituted by a set of “wait(completion semaphore)”. 

We shall use the cobegin . . . coend for it is syntactically 
more pleasing. 

The structural and behavioral models are in the same form 
as used in our top down system design descriptions. 


SLl CODE 



SE(@X0, @Y0,@X1, @Y1); 

FFT BUT(0X0, @Y0, 0X1, @Y1); 
XO+TSEOXO, FFT_BUT@X0); 

Y0+(SE0Y0, FFT_BIJT@Y0); 

XI + (SE0X1 , FFT_BUT@X1); 
Y1+(SE@Y1, FFT_BUT@Y1); 


SLl SOCKET HAP 


SOCKET 

EXTERNAL IDENTIFIER 

SE@X0 

XO 

SE@Y0 

YO 

SE@X1 

XI 

SE@Y1 

Yl 

FFT BUT (a XO 

XO 

FFT BUT9Y0 

YO 

FFT~BUT@X1 

XI 

FFT BIITPYl 

Yl 
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MACHINE PROCESSABLE SL1 MODEL 

Figure 7 shows a graphical representation of the GMB 
model of FFT BUT and of its standard environment SE. 
The structural model SL1 code and the socket to external 
identifier mapping are included in the figure. 

MACHINE PROCESSABLE BEHAVIORAL MODEL 

In similar fashion to the MESSAGE TRANSMITTER 
example above the building block model includes, for it and 
its standard environment, a control graph description, a data 
graph description, a PLIP interpretation for each processor 
and dataset, a socket behavior model and a mapping from 
GMB to SL1. If space permitted, the data graph interpre¬ 
tations would show how the SE processors PI and P2 pro¬ 
duce the real and complex inputs XO and YO respectively 
and write them in the named datasets. The initiated multi¬ 
plication process would be shown to use the input data and 
be followed by concurrent addition and subtraction. The 
real and complex results, XI and Y1 respectively, would be 
shown to be delivered to the named SE data sets completing 
the operation. The full model has been executed and 
checked in the SARA simulator. 

Additional behavioral characteristics of the model could be 
included. A memory requirement model might specify a 
bound of 1000 bytes. A precision requirement might specify 


that the outputs have a precision to >5 decimal digits. A 
timing model might specify a maximum time of one milli¬ 
second for each transformation. 

COMPOSITION 

We shall try now to describe composition of the FFT 
module using an ensemble of FFT Butterfly building block 
models to implement the algorithm defined by Cooley and 
Tukey. 30 

For an n=8 point FFT we have three basic alternatives 
for arranging the butterflies, in increasing order of parallel¬ 
ism, namely: 

(a) one butterfly performing the computation in (n*log(n))/ 
2 (=12) butterfly steps. 

(b) n/2 (=4) butterflies performing the computation in 3 
butterfly steps. 

(c) n/2*log(n) (=12) butterflies performing the computa¬ 
tion in one butterfly step. 

We chose the third alternative, for it yields the most 
parallel (and more interesting for our demonstration pur¬ 
poses) algorithm. 

At the time of writing this paper the composition proce¬ 
dure for taking building block models out of SARA’s library 
and linking them together automatically had not yet been 



Figure 8—Control graph for 12 butterfly FFT 
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implemented. Each step was executed independently with 
existing tools because it uses the same procedure as would 
be used when creating an original model. The final result for 
the FFT model is shown in Figures 8 and 9 which display 
copies of the building block model interconnected to create 
a concurrent system implementation of FFT. A flow chart 
of the composition procedure is given in Figure 10. 

Let us now consider the evaluation process. The flow-of- 
control is properly terminating. It can easily be abstracted 
back into a single control node because it is acyclic and its 
internal control logic is all “AND.” 

The data graph interpretation (PLIP) permits us to test and 
evaluate the FFT system interactively through the GMB 
simulator. The complete interpreted model was successfully 
executed. We could enlarge the design to cover a full signal 
processing system using the FFT as a building block so long 
as we were consistent with its ENV. 

What may be more important is that, in the same manner 
as we have shown for the MESSAGE TRANSMITTER 
example, PLIP can be reduced to PL1 code and we have 
completed a path from programming-in-the-large to pro- 
gramming-in-the-small. 

Obviously this is a demonstration problem which will get 
enriched as SARA matures. We do not presently propose, 
for each butterfly building block, a machine which can com¬ 
pile and execute PL1 programs. Such a machine may not be 


available on an LSI chip for a little while. In fact a principal 
goal of current SARA development is to provide interpre¬ 
tation of the data graph in languages which transform di¬ 
rectly to programs that execute on microcomputer building 
blocks residing in a SARA system architecture environment. 

CONCLUSIONS 

As a superstructure to be imposed upon the final code or 
software systems, the structural and behavioral models have 
the following objectives: 

• To show the potential flow of resource names between 
the structural components of a software system. 

• To give a designer complete control over the distribu¬ 
tion of information between modules. Sockets are the 
place where one states one’s intent with regards to 
identifier visibility and access rights among modules at 
any level. The sharing of resource names is facilitated, 
without unwanted spread of accessibility, that is, dif¬ 
ferent types of access are distinguishable (read-only, 
read-write), and access to different subsets of a set of 
resources can be specified. The idea here is to enforce 
the specifications over the source code at compile/link 
time. 
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• Both top-down and bottom-up design strategies are sup¬ 
ported by multilevel modeling facilities. 

• Hierarchical organization is enforced, and the structural 
model, in conjunction with the flow-of-control and flow- 
of-data models, can keep around conceptual entities 
that would otherwise vanish in the design process. The 
whole design tree is kept, not just the leaves, thus 
making the final product comprehensible both top-down 
and bottom-up. The SL1 model and the list of attributes 
associated with each of its sockets encompass what has 
been proposed in the literature 5,34 as a MIL (Module 
Interconnection Language). Our approach is more gen¬ 
eral in that we can define and enforce interface com¬ 
mitments among concurrent software systems. 

• To increase the validity of the final product, all map¬ 
pings and checks for completeness and consistency can 
be done with machine assistance through SARA. 

With respect to flow-of-control, for example, SARA pro¬ 
vides analysis facilities to check for proper-termination. A 
properly terminating flow-of-control model is reentrant, 
deadlock-free and has determinacy, i.e., regardless of the 
relative order of execution of the subprocesses involved, the 
unique output arc is reached and, aside from the vacated 
entry arc, the state of the control graph is the same as it 
was initiated to. 

The main contribution of the SARA system for software 
synthesis is the fact that it offers a continuous path from 
programming-in-the-large all the way down to programming- 
in-the-small of possibly concurrent systems, while providing 
a designer with interactive tools and checking procedures to 
enforce consistency between requirements, structure, func¬ 
tion and behavior. 

There are a number of issues which require further study: 

• The statement of requirements of a design is the cor¬ 
nerstone of the use of the design methodology. So far 
the English language description of this set is used in 
SARA. A more formal scheme for describing require¬ 
ments is needed, as well as the associated criteria to 
evaluate whether or not a given design satisfies the 
previously stated set of requirements. 

• The structural and behavioral models cited in this paper 
(SL1 and GMB) are restricted to representing static 
topologies. We currently handle any dynamic behavior 
in the interpretation associated with the data graph (i.e., 
in PLIP). Many interesting design problems, especially 
in the area of operating systems could be represented 
in a richer form if we could model and smoothly inter¬ 
face with a modeling scheme in which both the structure 
and the topology of the behavioral models could change 
with time, during execution. Other desirable features 
would be the ability to model recursion, dynamic stor¬ 
age and process allocation (i.e., everything requiring 
unbounded amounts of a resource) at the level of the 
GMB control and data graphs. 

• Refinement and abstraction rules for GMB primitives 
are proposed in Reference 27, but they are fairly re¬ 


strictive. The difficult problem here is to preserve 
“equivalence” between the two adjacent levels, both 
during refinement and abstraction. 

• Many interesting asynchronous systems, especially in 
the process-control area are interrupt-driven. The mod¬ 
eling of interrupts in a GMB generates at least two 
control arcs per interrupt, which tends to overcrowd 
the control graph. Some extension to the current work¬ 
ings of the token machine should allow for a simpler 
way of modeling perhaps a restricted class of interrupt 
handling schemes, such as only allowing one interrupt 
handler for all interrupts. 

• Currently, the only way two GMB processors can trans¬ 
fer information between them is through the use of 
datasets, which act as external (global) variables for the 
processors involved. We would like to provide design¬ 
ers with a means of passing parameters between GMB 
processors (although we might implement them inter¬ 
nally in SARA as global variables) in order to make the 
process of translation from model to code easier. 
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GM network station—A low cost 
graphics system for body tooling 

by THOMAS J. RENO 
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Warren, Michigan 


INTRODUCTION 

Over the years, General Motors has been actively involved 
in both the development and use of Computer Aided Design/ 
Computer Aided Manufacturing systems to aid in the design, 
engineering and manufacturing of automobiles. The use of 
CAD/CAM technology permeates most of our operations, 
but nowhere is it as important and pervasive as in the body 
tooling process. The demands for safety, environmental pro¬ 
tection and energy conservation have required a respon¬ 
siveness that only CAD/CAM systems could provide. No 
longer can we afford the long lead times of 3+ years pre¬ 
viously required to go from design concept to steel. Today 
we must respond quickly and effectively to an ever changing 
car market—and CAD/CAM is helping us to do so by dra¬ 
matically shortening the body tooling cycle. 

The impact of CAD/CAM on our body tooling process 
has been demonstrated many times, but never more con¬ 
vincingly than in the years following the 1974 oil embargo. 
This new technology enabled us to produce the 1976 sub¬ 
compact Chevette in only two years in response to a then 
rapid shift of public sentiment to small cars. 

CAD/CAM contributed heavily to the design and tooling 
of our 1977 full size cars. In fact, Mr. Howard Kehrl, GM 
Executive Vice-President, stated in a Business Week article 
that only through use of this technology were we able to 
produce our 1977 ‘B’ and ‘C’ models at all, after a nine 
month initial delay and a major shift in specifications half¬ 
way through the program. Computer graphics was also used 
to evaluate the structural characteristics of proposed designs 
and to eliminate much of the trial-and-error process previ¬ 
ously involved in making design changes. 

CAD/CAM has truly come of age and is now an integral 
part of our business. It is difficult to imagine how we could 
design and build a car without it. But CAD/CAM did not 
just happen in General Motors. Its acceptance today is the 
culmination of a long process begun in the late 1950’s when 
the possibilities of the newly emerging technologies of Nu¬ 
merical Control and visual man-computer interaction were 
first recognized. 


SYSTEMS ORIGINS 

The GM Research Laboratories pioneered man-computer 
communication through their DAC (Design Augmented by 
Computer) project. At the same time, GM Manufacturing 
Development was developing N/C technology for sculptured 
body tooling. Significant resources of both people and 
money were devoted to these projects and to related devel¬ 
opments in the areas of mathematics and computer science 
until usable prototype systems were available. These pro¬ 
totype systems included sophisticated new mathematics ca¬ 
pable of representing the types of curves and sculptured 
surfaces found on automobiles; data bases and data struc¬ 
tures capable of storing these mathematical models; regional 
milling capabilities for both ball cutters and end-mill cutters; 
and user interfaces which permitted the easy addition of 
other new applications such as structural analysis, vision 
studies, etc. 

In 1967, Manufacturing Development was assigned the 
responsibility of transforming these prototype systems into 
usable production tools. The result was two complemen¬ 
tary systems, CADANCE (Computer Aided Design and Nu¬ 
merical Control Effort) and INCA (Integrated Numerical 
Control Approach), which together provided an effective 
CAD/CAM base for design, engineering and tooling. Addi¬ 
tional application and system development was undertaken 
by Fisher Body and Design Staff to finally bring all of the 
development efforts into full production use. The systems 
in use today at GM are the result of all these combined 
efforts. 


COMPUTER AIDED DESIGN BACKGROUND 

The CAD systems in General Motors were all developed 
as outgrowths of the original DAC system and used com¬ 
puter graphics and time sharing as the base upon which they 
were built. Today approximately 75 graphic consoles served 
by two large computer installations at the GM Technical 
Center are employed by our divisions and staffs to do body 
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design and engineering work. The CAD systems all employ 
refresh graphic display devices and accept virtually all of 
their input via light pen selection from the face of the screen. 
Similarly, all of the output is transmitted to the user via 
displays on the screen. This combination of time-sharing, 
graphic input and graphic output provides an unbeatable 
combination for problem solving. Mistakes are detected in¬ 
stantly and the problem can be re-run at once. Long delays 
waiting in input and output queues are totally eliminated. 
The lead time savings achieved using this technology are 
truly impressive. Very often 6-8 weeks are eliminated from 
a conventional 12 week job. 

COMPUTER AIDED MANUFACTURING 

BACKGROUND 

The CAM systems developed for body tooling exploited 
the technology of Numerical Control and used the computer 
facilities available at the time. This was, as mentioned ear¬ 
lier, in the late 1950’s and 1960’s. At that time, batch mode 
computer runs using card input and obtaining printed output 
was the standard bill of fare. Some minor improvements 
such as Remote Job Entry (RJE) facilities and Time Sharing 
Option (TSO) were added to the batch systems, but the 
basic philosophy of our CAM programs remained un¬ 
changed. There was no way to avoid the long waits in input 
queues and output queues, the randomness of courier deliv¬ 
eries, or the large stack of printed output which had to be 
examined for possible errors. 

The CAM programs, such as INCA and APT, used by 
GM were developed to process the masses of numerical 
control data required for wood model and die construction. 
Human decisions were emulated within these programs 
wherever feasible or possible. Nevertheless, human inter¬ 
vention was required to analyze and evaluate intermediate 
results produced by these computerized processing aids. 
These results were in the form of computer listings and 
usually had to be converted to graphical form via sketches 
or drafting machine drawings. This process obviously was 
very time consuming. 

For a number of years, this mode of operation was ac¬ 
cepted, although the long time delays associated with each 
step were excessive. The technology of N/C, however, pro¬ 
gressed and saved time and money in spite of long process¬ 
ing times. Indeed, instead of working to improve the proc¬ 
ess, we worked on improving the technology of N/C. 
Regional milling techniques and flow line cutting were honed 
to perfection. Sophisticated evaluation techniques and sur¬ 
face linking programs were developed and brought into pro¬ 
duction use. A data storage and retrieval program which 
virtually eliminated data cards was perfected and has been 
operational since the late 1960’s. CAM technology indeed 
grew to where today nearly all full size interior and exterior 
sheet metal wood models are cut using N/C. 

REASONS FOR CHANGE 

All the while, however, the operational procedures re¬ 
mained unchanged. The CAM systems became a victim of 


the computer technology upon which they were based. The 
batch mode operating systems did not allow any meaningful 
operational improvements to be made. It still took several 
weeks to produce a complete set of cutter tapes for a com¬ 
plex wood model. The CAD systems, on the other hand, 
had benefited from new advances in computer hardware 
and software and were achieving remarkable results in both 
processing capability and lead time reduction. The N/C lead 
times began to be the bottleneck and it soon became ap¬ 
parent that improvement in processing techniques was es¬ 
sential. 

Based on the effectiveness that computer graphics had 
demonstrated in the area of Computer Aided Design, it was 
obvious that this technology could provide similar process¬ 
ing improvements to our Computer Aided Manufacturing 
systems. However, it was not obvious exactly how to pro¬ 
ceed to implement a graphics CAM system in GM. Since we 
were attempting to modernize an existing operation—not 
start a new one—we had to take into account constraints 
the existing program imposed on us. These were geography, 
cost, and time. 

GEOGRAPHICAL CONSTRAINTS 

In GM, the vast majority of design activities occurs within 
the square mile area of the GM Technical Center. Tool 
manufacturing, however, encompasses a much larger area 
and even extends to West Germany where Adam Opel is an 
active participant in our CAM activities. Any modification 
of our CAM system for body tooling had to encompass all 
tool manufacturing. Computer graphics—even today—has 
difficulties economically locating consoles more than one 
mile from the computer. Consequently, any move of our 
CAM systems towards computer graphics had to be inde¬ 
pendent of the large Technical Center computers. 

COST CONSIDERATIONS 

As to cost, the batch N/C systems had been tuned over 
the years to the point where processing charges were quite 
minimal. Also, CAD via graphics had created an excellent 
data base from which our CAM activities could begin. This 
meant that very often the full power of the large Technical 
Center computers was not needed for our CAM work, and 
if we used them we would be incurring greater processing 
costs than necessary. This again indicated that a solution 
independent of the large Technical Center computers was 
required. 

IMPLEMENTATION TIME CONSIDERATIONS 

The repertoire of computer programs used in die model 
construction is very extensive. All of the programs are re¬ 
quired in some degree to complete the job. Any moderni¬ 
zation of our CAM system would have to take this into 
account and provide access to similar facilities. Since it 
would be impractical and very expensive to convert all ex- 
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isting programs to a graphical format, any new system would 
have to be compatible with the in-place system. 


SYSTEM SPECIFICATION 

With these considerations in mind. Manufacturing Devel¬ 
opment began exploring hardware/software combinations 
that would meet our objectives. We needed a system that 
was inexpensive, could operate independently of large com¬ 
puters, and yet could communicate freely with the existing 
in-place systems. In 1973, we began development of a Low 
Cost, Stand-Alone NIC Review System to bridge the gap 
between CAD and CAM. This system answered the ques¬ 
tions of geography, cost and time and seemed to be a start 
towards getting the benefits of graphics into our body tooling 
manufacturing. 


HARDWARE COMPONENTS 

The system developed consisted of: 

Mini-computer with Disk Storage 
Interactive Graphic Terminal 
Medium Speed Dial Phone Line 
N/C Tape Punch 

Its total purchase price is approximately $75,000, which 
equates to less than $ 10/hour for a one-shift operation. 
This is well below the cost of large central time-sharing 
computers and even less than batch system costs. The 
system is indeed a stand-alone system. It can be placed 
in any room of any plant and requires no special wiring, 
floors, or unusual cooling. Its dial-up phone line and 
N/C tape punch enable remote units to communicate with 
each other or with the central computers at the Technical 
Center. These features overcome the geographic and imple¬ 
mentation time constraints imposed on our CAM system. 


HISTORY AND INITIAL IMPACT 

Initially, this system simply provided a graphic input and 
output facility for our existing batch mode CAM programs. 
Substantial lead time was gained by simply reviewing data 
and input commands graphically before submitting them, 
thereby detecting errors early in the game and correcting 
them before batch runs rather than after. Similarly, graphic 
review of the output was faster than laboriously scanning 
pages of printout or waiting for full size drawings to be made 
on already overloaded drafting machines. 

The initial components of the system were few. There 
was a data base manager to store and retrieve information. 
Function buttons were available to change views, shift, 
scale, rotate data, etc. 

In addition to these graphic data review features, other 


computational programs were available to: 

Create arcs, circles, straight lines 
Modify Data (delete, insert, move points) 

Transform Data (Rotate, Translate, etc.) 

Create Card Images for Job Submission to Large Com¬ 
puters 
Teleprocess 
Punch Mylar Tape 

This simple review system allowed us to export Computer 
Aided Manufacturing technology to our plants and was a 
start towards bringing the benefits of computer graphics to 
the manufacturing floor. 


A CHANGE IN DIRECTION 

The use of this low-cost review system never became very 
pronounced, however, because even while doing our initial 
development work on the system we realized the potential 
we had at our fingertips. It became readily apparent that the 
mini-computer was never really being taxed. The users also 
posed the question, “Why, since I have all my data here, 
can’t I generate my N/C tapes here?” This, then, marked 
the turning point of the project and brought about the re¬ 
naming of the system to the ‘Network Station.’ This name 
symbolizes two things. First, the system is still part of our 
overall CAD/CAM program and depends on access to the 
central data bases via the telephone network. Second, once 
the data is received, the system operates as an independent 
station until the job is complete. The term ‘Network Station’ 
seems to fit nicely the function and use of the system. 

DEVELOPMENT AND CURRENT CAPABILITIES 

User divisions enthusiastically accepted this change of 
direction and even contributed manpower to assist in the 
software development. Eight departments representing five 
divisions and staffs have participated in the Network Station 
development. Some of the items added to the software rep¬ 
ertoire to make possible a full stand-alone operation include: 

Additional Curve Generation Functions (Fillets, Sweeps, 
etc.) 

Curve Smoothing 
3-D Meshing 
Surface Development 
Surface Linking and Trimming 
N/C Machining 

Interactive Cutter Location Verification and Modification 

Through the use of this system, it is now possible to com¬ 
pletely process our most difficult wood models without using 
any of our old batch CAM programs. This includes surface 
definition, cutter location, and N/C tape punching. Divisions 
with easy access to the Technical Center computers still 
find it more convenient to use the existing old CAD/CAM 
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interfaces, but they are all acquiring Network Stations for 
future use. Someday this new graphic CAM system may 
make our old batch systems completely obsolete. 

NETWORK STATION DATA FLOW 

The Network Station offers a range of CAM uses never 
before available in our body tooling process. The ability to 
teleprocess information, read in punched tapes, or create 
data directly on the system provides for a wide variety of 
different input sources. For example, inputs from all of these 
sources are a commonplace occurrence today. 

Teleprocessed Information from CAD Data Bases 
Digitized Information on Punched Tape 
3-D Input from Hard Model Scanners 
Cutter Location Information on Punched Tape 
Experimental Drawings 

These input sources cover all of the common ways in which 
data is now collected. Once entered into the system, the 
user can proceed to process the model. When finished, a 
number of choices are available for outputting results. These 
include: 

Punched Tape for Computer Controlled Mills 
Punched Tape for Drawing Machines 
Data for Teleprocessing 

(a) To a Large Computer for Additional Processing 

(b) To a Remote Site for Machining, Viewing, etc. 
Printed Process Sheets 

Printed Checking Information 


NETWORK STATION ACCOMPLISHMENTS 

As a testament to the productivity and flexibility of the 
Network Station, Fisher Body, Die Engineering Activity, 
now uses this system for virtually all of their CAM work 
related to body tooling. They have six systems which they 
now use to produce: 

All Wood Die Models (Inners & Outers) 

All Seat Bun Patterns 
All Bend and Sweep Gages 
Production Glass Block Tapes 
Die Profile Tapes 

DIVISIONAL INSTALLATION 

Fisher Body is by far the largest single user of our Net¬ 
work Station. They are not, however, the only user. Seven 
divisions and staffs have equipment either installed or on 
order. Several others are actively pursuing authorization for 
equipment. The tally as of now is 18 units installed, 3 await¬ 
ing delivery and 3 in the process of approval. The divisions 
also have tentative plans for 10-12 additional units in the 


next two years. These units are located as far away as West 
Germany, thus demonstrating the true stand-alone nature of 
the system. 

GROWTH OF CAD/CAM CENTER 

As you can see, the Network Station is rapidly becoming 
the dominate force in our N/C body tooling system. The 
growth of this system is well in tune with the expanded use 
of our Computer Aided Design programs and indeed com¬ 
plements and supplements our design and engineering sys¬ 
tems. As the use of CAD grows, it too is spreading out 
geographically to our outlying divisions and plants. We are 
even now locating graphic design consoles in Lansing, Pon¬ 
tiac and Flint. The hardware used for both our CAD and 
CAM systems is interchangeable and thus permits multiple 
use of the same equipment. 

MULTIPLE USES OF EQUIPMENT 

For example, a typical remote installation consists of: 

Mini-computer and Peripherals 

Graphic Console 

Drafting Machine 

When the mini-computer and the Graphic Console are 
connected, the result is a Network Station. The Design 
Console can be connected over high-speed telephone lines 
to the central CAD computers, thus allowing the mini-com¬ 
puter to be used for teleprocessing, other computing appli¬ 
cations, or as an input source for the drafting machine. 

The drafting machine, of course, can also operate inde¬ 
pendently of the mini-computer. This combination of hard¬ 
ware and software allows full use of all equipment, provides 
access to both CAD and CAM programs, and allows the 
user to operate either as part of the central design and 
engineering community or as a stand-alone station. Also, 
since no specialized hardware is used, it is easily maintained, 
can be inexpensively upgraded for more power, and a wide 
range of peripherials such as plotters, digitizers, etc., can 
be easily added. 

ADAPTABILITY OF SOFTWARE 

The intent of the Network Station has always been to get 
some computer power out to the plants where the work is 
actually done. Realizing that each plant operates somewhat 
differently, and knowing that one central development ac¬ 
tivity could never fill all of the needs of all the plants, we 
designed the Network Station system to be very easy to 
program. All application programs are written in Fortran. 
All input, output, and graphic commands are available to 
the programmer via simple subroutine calls. A simple pro¬ 
grammer’s manual is available which describes how to write 
Network Station programs. In addition to our own staff, we 
have trained twenty-two people representing six divisions 
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and staffs how to add applications to the system and they 
have all contributed to some degree. Our experience has 
been that a programmer knowledgeable in Fortran can be 
an accomplished Network Station programmer in a period 
of approximately four weeks. 

POTENTIAL FOR COMPUTER AIDED 

MANUFACTURING ~ 

The impact of computer graphics on the GM body tooling 
cycle has been very substantial. But full lead time and dollar 
savings will not be achieved until the entire body tooling 
cycle from design concept to production die is affected. To 
date the major impact of this new technology has been in 
the area of product design. Our large graphics systems have 
been profitably used in body design and body engineering 
for several years. However, examination of the distribution 
of work in the body tooling cycle shows that 88 percent of 
all hours are expended in tool design and construction. Sim¬ 
ilarly, 70 weeks are consumed in this effort as opposed to 
only 17 weeks for panel design. Clearly, the greatest poten¬ 
tial savings are in these areas, and we have scarcely begun 
to impact them. 

THE BODY TOOLING SCENARIO 

What is ultimately required is a general machining-man¬ 
ufacturing system for body tooling. This system would use 
high performance Numerical Control machines to do the 
milling and inspection of die models and dies. The process¬ 
ing, communications, and coordination would be done on 
small computers linked together by telephone lines. The 
system we envision would consist of at least three major 
parts. 

The first would be Tool Design. Here the die models, dies 
and fixtures would be designed. The product design data 
would be used to determine how many and what kinds of 
dies are needed to produce a part and also the sequence in 
which the dies would be used. The amount of die tipping 
would be determined and the runoff conditions developed. 
Binders and die and checking fixtures would be designed. 
All the necessary drawings would be output. 

The second part of the system would deal with the ma¬ 
chining of the die models, dies, and related tools. The prod¬ 


uct design and tool design data would be used to generate 
N/C tapes to drive the mills. These tapes would contain all 
the machining information. They would be verified and used 
to machine wood die models, gages, and glass blocks. Tapes 
to cut metal would be produced either from the design data 
or by tracing the wood die models with computer controlled 
scanners. All tapes, process sheets, and listings that are 
necessary for setup and milling would be produced. If the 
use of computer controlled inspection machines becomes 
widespread, these tapes would also be generated. 

The third part of the system would ensure proper utili¬ 
zation of equipment and metal in the stamping plants. Offal 
would be scrutinized to see if other parts could be produced 
from it. The press configurations and material handling 
equipment would be checked to see if they were available 
and not overloaded. Once this was assured, layout drawings 
would be generated and sent to the plants. In addition, 
material deviation studies would be done to ensure that no 
plant was using a higher grade metal to produce a part than 
other plants. 


CONCLUSION 

We are slowly working towards developing such a system, 
and the Network Station plays an important role in these 
future plans. We are already actively working with the di¬ 
visions to develop programs in the areas of die processing, 
press planning, clay milling and 5-axis tool center compu¬ 
tations. A prototype program for generating N/C inspection 
tapes is already available on the Network Station. Also, 
programs to minimize metal waste through optimum blank 
nesting are now under way. We are convinced that for N/C 
technology to succeed, it must become the “conventional 
process.” The Network Station is helping this happen by 
placing the computer in the plant where the work is done 
and by removing the mystique from Computer Aided Man¬ 
ufacturing. 

In summary, we have a tiger by the tail in our Network 
Station project. He has “devoured” every task given to him 
and still seems to have a voracious appetite. We are letting 
go of the tail a little at a time as we assign more and more 
of the new body tooling applications to him. We feel that it 
will take quite a while before either his appetite is satisfied, 
or we “run out of tail.” 





Computer aided design (CAD)/technical 
data automation (TDA) 


by VERNON R. PEARL 

U.S. Army Research and Development Command 
Aberdeen Proving Ground, Maryland 


INTRODUCTION 

The Technical Data Package (TDP) is the terminology used 
by the U.S. Government to describe the documentation 
generated during research and development for the procure¬ 
ment of end items. 

Spiraling costs in producing TDP’s have long been a major 
concern of the Department of the Army. Thousands of en¬ 
gineering drawings, specifications, standards, and other re¬ 
lated technical data that make up a TDP are produced each 
year. These TDP’s are used to document, in detail, the 
Army materiel end items required to supply troops in the 
field and to maintain a state of constant military readiness. 
With industry and Government continuing to change the 
technological base in both design and manufacture, keeping 
documentation up to date has become costly and difficult. 
Because of the continually changing technology, end items 
may soon become obsolete and many of the repair parts 
required to maintain fielded items would no longer be man¬ 
ufactured. In either case, it’s “back to the drawing board” 
to start a new design, improve an existing one, or rework 
items in the field to allow for procurable spare repair parts. 

At the CSL-Technical Support Division, ARRADCOM, 
which services the CSL at Aberdeen Proving Ground, Mary¬ 
land, there are approximately 50,000 engineering drawings 
on chemical-related items. There is also a vast number of 
standards and specifications associated with these chemical 
items. New chemicals are constantly being developed. 
Sometimes the specific chemical required to produce a cer¬ 
tain munition is not always available. These factors contrib¬ 
ute to a high rate of both drawing and specification changes 
and lead to many contract waivers and deviations. All of 
these problems result in changes to the technical data and 
higher documentation costs. It is for these reasons that we 
begin looking into using the computer in an effort to reduce 
both time and costs and to streamline the entire technical 
data generation process. 

EXISTING EQUIPMENT AND SOFTWARE 

The equipment selected for accomplishing computerized 
engineering drafting is as follows: 


1. A plotting and digitizing system (manufactured by Actron 
Industries). 

This system was purchased in 1972. It consists of a 5- by 
12-foot drafting table and a control console containing a 
digital computer and other electronic equipment related to 
monitoring and measuring functions. In addition, the console 
is equipped with a closed-circuit TV monitor providing the 
operator with a magnified view of the drawing area and 
enabling him to select individual points for digitizing or to 
monitor and supervise an automatic drafting operation (Fig¬ 
ure 1). A drafting and digitizing head suspended from a 
gantry and armed with an optical line-following device pro¬ 
duces the X-Y coordinate data required for line definition. 
The system is basically restricted to two-dimensional work 
but certain perspective, isometric, and three-dimensional 
views can also be produced. In the digitizing process (the 
ability to automatically define lines and curves), the meas¬ 
urement system continuously reports line locations to the 
computer. The computer is programmed to process this data 
into a tool-path description and output a numerically-con¬ 
trolled tape complete with spindle speeds, feed rates, and 
other miscellaneous functions. The completed tape can be 
inputted directly to the machine tool without any further 
need for processing.* 


2. A refresh interactive graphics system (manufactured by 
Information Displays Incorporated). 

This stand-alone system was purchased in 1973. It con¬ 
tains a display console, a display controller, a graphic dis¬ 
play processor, a teletype station, a magnetic tape unit, and 
a disc. The display console consists of a 21-inch cathode- 
ray tube (CRT), a light pen, and an alphanumeric keyboard. 
The display controller has vector, character, and circle gen¬ 
erators, four types of line structures, and four levels of 
intensity control. The graphic display processor consists of 
32K of core. It has hardware multiply/divide, direct memory 
access, a priority interrupt module, and an automatic boot 


* SME Technical Paper MS73-958, “Application of Automated Drafting and 
Digitizing at Edgewood Arsenal,” by Vernon R. Pearl. 
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Figure 1—Drawing being digitized 


strap loader. The magnetic tape unit uses 7-track, 800-bpi 
dual-density tape. The disc has a 1.1 million word capacity. 
The drafting and design software consists of drawing lines, 
circles, arcs, and rectangles. Texts and symbols can also be 
generated. Other functions include move, plumb, level, 
copy, delete, and group. The system also contains library 
support for storing symbols and procedures for saving, start¬ 
ing, getting, and deleting drawings. The system is a general- 
purpose operating system programmed in Fortran. In 1975, 
an additional display console was added to the basic system 
including a time-share capability which allows for simulta¬ 
neous and independent utilization of either console (Figure 
2). In 1976, a third stand-alone system was purchased and 
is currently being interfaced with the laboratory’s Uni vac 
1108 computer. Other software improvements recently 
added include three-dimensional modeling/dynamic simula¬ 
tion and printed circuit-board-layout routines. 

These three systems in conjunction with the drafting/dig- 



mBlm 


teiisiii 



iipgii* 



liiliSilSlli 



liisiiiiiiii 

,,, : / • 
wM § ''s ' /t - 


gillliiillffisiil 














CAD/TDA 



Figure 4 —Drawing details 
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Figure 5—Assembly drawing 


itizing system have given us an extremely powerful com¬ 
puter-aided-design system.* 

COST-SAVING EXAMPLES 

There are many varied cost-saving projects that have been 
accomplished through the use of this computer-aided-design 
system. The examples chosen were picked mainly because 
of their relation to technical data work. Keep in mind, how¬ 
ever, that printed circuit boards, numerical control, floor 
plan layouts, etc., can also be accomplished using CAD. 

1st example 

This example is one of the more common types performed 
with the system. An engineer brought in a sketch of a binary 
munition (Figure 3). He had semi-scaled the sketch to get 
the proper proportions. With this information we were able 
to digitize the various proponents using the Actron equip- 

* SME Technical Paper MS77-764, “Cost Effectiveness of Computer-Aided 
Design (CAD),” by Vernon R. Pearl. 


ment and ascertain such data as volume, first moment about 
the Y axis, center of mass along the X axis, second moment 
about the X-Y axis, radius of gyration, polar moments, and 
surface areas. The significance of using this program is that 
while you are designing you can easily determine moments 
at any time without using any mathematical formula or doing 
any computer programming. A density factor can be input 
through the teletype to allow calculations to be made of an 
object or shape which is constructed of nonhomogeneous 
material in seconds. The design can be immediately adjusted 
to achieve the configuration necessary for proper flight sta¬ 
bility. Once the engineer is satisfied with the design config¬ 
uration, the parts are automatically drawn up on the CRT 
(Figure 4). The parts are then combined to make an assembly 
drawing (Figure 5). This combining operation checks for 
both alignment and proper fits, such as does the O-ring 
engage properly to its mating part. The system also has the 
ability to automatically dimension any line or circle that has 
been put on the screen. This method of design allows the 
operator to make changes rapidly, catch errors, save on 
fitting calculations, and merge drawings. 

Twenty prototype drawings were required to describe the 
items sufficiently enough for fabrication by the experimental 
metal shops. By using the computer-aided-design system on 
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this job, all calculations and drawings were completed within 
two weeks. It is estimated that approximately two months 
would have been required if it had been done manually. 

2nd example 

This area of cost savings comes from the use of a three- 
dimensional software package with dynamic rotation that 
resides within the graphic system. The Research Division of 
the Chemical Systems Laboratory uses this software routine 
to visualize and verify molecular binding. Before this routine 
can be used, certain coordinate data describing the molecule 
must be fed into a molecular coordinate calculation program 
that resides within the Laboratory’s Univac 1108 computer. 
This program gives the scientist an idea of the shape of the 
molecule and produces a printout yielding the X, Y, Z co¬ 
ordinates of the various elements associated with it. This 


data is then fed into a molecular orbital calculation program 
which produces X, Y, Z coordinates for a minimum energy 
configuration of the molecule. The scientist has no way to 
visualize these several sets of data in space. Incorrect data 
fed to the computer would go undetected. Erroneous data 
used in further research proved costly in both time and 
manpower. 

To eliminate this problem, the initial molecular coordinate 
data is now displayed on the graphics system after the initial 
Univac 1108 data is received (Figure 6). Here it is possible 
to rotate the entire molecule at any atomic point and at any 
angle to check the bonding for errors. This verified data is 
then fed back into the Univac 1108’s molecular orbital cal¬ 
culation program. The new data is again displayed on the 
screen. This becomes quite significant, since the scientist 
can now see the minimum energy configuration which may 
bear little resemblance to the original configuration. This 
visual display allows for a more direct and rational evalua- 



Figure 6—Molecular rotation displayed on the cathode-ray tube 












348 


National Computer Conference, 1978 


tion and replaces expensive manual simulation methods. We 
also have the most probable molecular configuration which 
will subsequently be used in various biological and physical 
studies. Though the savings are somewhat intangible, one 
can easily see that by using the computer-aided-design for 
just one hour per molecular validation many man-hours of 
the researchers labor are continually being saved. 


3rd example 

This example is a mechanical schematic, which was ac¬ 
complished with ease by using the graphic system. A free¬ 
hand sketch giving an idea of the flow was the only require¬ 
ment needed to get started. Symbols generated months ago 
and stored were pulled from the library and displayed on the 
screen. New symbols were generated as required. The 
plumb and level functions were used to line up these sym¬ 
bols. A line connecting function was used to tie the symbols 
together. The alphanumeric keyboard was used for labeling. 


Once again changes were completed rapidly. Sections could 
be grouped and relocated anywhere in the drawing in sec¬ 
onds (Figure 7). 

Including changes, the drawing was completed in one day, 
and it is estimated that three days would have been required 
if it had been done manually. Electrical schematics and 
wiring diagrams are also easily handled using the same tech¬ 
niques. 

4th example 

The final example depicts a set of production drawings 
used in a TDP for manufacture by contract. To prepare 
drawings for a TDP was the primary reason the equipment 
was purchased. The development of a TDP can take any¬ 
where from two to six years and requires many man-hours 
of labor and various prototype hardware building and testing 
methods. Engineering drawings, and changes to them, are 
involved in nearly every step of the development phase. The 
cost of producing drawings while going through this cycle. 
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has continually increased. When the prototype phase is com¬ 
pleted, the drawings must be redrawn according to DOD-D- 
1000 in a level 3 format. This format is required for all 
engineering production drawings associated with a TDP. 
The final drawings that become a part of a TDP must be 
complete and clear so that the manufacturer has no doubt 
about what he is to produce. Eight prototype drawings orig¬ 
inally done on the graphics system were required to be 
converted to production drawings. The number of drawings 
was expanded from eight prototype drawings to 22 produc¬ 
tion drawings to meet the military specification. Drawings 
had to include subassembly, packing, marking, and individ¬ 
ual details on each part (Figures 8, 9, and 10). Careful in¬ 
spection of the drawings illustrated in the first example with 
the examples shown above will show the difference between 
prototype and production drawings. By using the computer- 
aided-design system, the job was completed in three days. 
If this work had been accomplished manually, at least three 
to four weeks would have been required. After the initial 
drawings were made, an engineering review was conducted 
which resulted in many additional changes. Using computer- 
aided-design, all the drawings were made in one day. One 
week would have been required if done manually. 


These examples are only a few of a large variety of en¬ 
gineering applications that can be accomplished through 
computer-aided design. These examples show simple uses 
in computer-aided design illustrate that it can be an excellent 
investment. The average manpower savings generated is 
about a 4 to 1 ratio. The equipment purchased were both 
turn-key systems. The drafting and digitizing system cost 
$250,000 and was amortized in less than 2 Vi years. The first 
interactive graphics system costing $150,000 was amortized 
in 1 Vi years and the second system, worth $75,000, showed 
a return on investment in less than one year. With these 
results in equipment and software amortization, there is little 
doubt that computer-aided design has become an economic 
necessity. 


TECHNICAL DATA AUTOMATION 

The automation of Technical Data is concerned with a 
means of creating, storing, and retrieving technical data 
using computer technology. Within the Department of the 
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Army, there is a requirement to develop and maintain var¬ 
ious types of data! This data is in the form of engineering 
drawings, specifications, standards, packaging sheets, and 
Quality Assurance provisions. All of this data is currently 
manually reduced from original drawings or type-written 
pages to 35mm film and mounted in 80-column aperture 
cards containing the document’s identification. These aper¬ 
ture cards are manually gathered and form the technical 
documentation required for government procurement. 

Computer-aided design and drafting techniques have made 
it possible to create an engineering drawing on a Cathode- 
ray tube (CRT). This method has proven to be extremely 
fast and highly cost effective. It also permits the user to 
create, store, and retrieve engineering drawings electroni¬ 
cally in digital form. 

This initial step has formed the basis for the automation 
of technical data. Data stored in this fashion allows us to go 
directly to a computer output microfilm unit to obtain the 
35mm film which is then mounted on the aperture card. 


Funds are currently available for such a unit and it is antic¬ 
ipated the system will be purchased and in operation by the 
end of 1978. 

A major problem, however, develops in trying to get ex¬ 
isting drawings and various data into the computer system. 
Manual digitizing is too costly. To eliminate this problem, 
a method is required to scan the drawing or card, vectorize 
the data and put it into computer memory. Negotiations 
have recently been completed between ARRADCOM and 
Altman Associates, Incorporated for an aperture card scan¬ 
ner. The major problem is to extract the data without de¬ 
stroying the inherent vectorization. This is to be accom¬ 
plished by the use of a high speed random access 
combination scanner and laser line follower operating with 
a dedicating computer (Figure 11). The system is expected 
to be in operation early in 1979. 

A mass memory storage unit will also be procured in the 
near future to handle all the data requirements involved with 
the technical data package. It will provide on-line storage 
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Figure 11—Optical schematic for scanner 
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TECH. DATA AUTOMATION 



A. CREATE DWGS 

B. REVISE DWGS 

C. REVIEW DWGS 

D. TD/CMS FILE 

Figure 12—Technical data automation flow diagram 


and retrieval capabilities of digital data in archival, non¬ 
erasable form and permit random access to any identifiable 
record. 

The combination of this equipment will enable us to 
change existing drawings or specifications, etc., through 
Interactive Graphics and go directly to aperture cards with¬ 


out the need of hard copy. The target date for the total 
system completion is set for 1980. When completed, this 
technical data automation system will save the government 
thousands of dollars and greatly reduce the engineering lead 
time now required in the development of Army materiel 
(Figure 12). 







The Boeing electronic computer 
aided design system 

by BRUCE H. INMAN 

Boeing Aerospace Company 
Seattle, Washington 


PROBLEM 

A large percentage of the cost of most DOD-contracted 
programs is concerned with the concept development, cir¬ 
cuit design, packaging, test and fabrication of electronic 
systems and subsystems. The application of Computer 
Aided Design (CAD) concepts to facilitate these operations 
can significantly reduce cost and minimize flow time in the 
design, fabrication and test of end item hardware. 


OBJECTIVE 

The fundamental objectives of CAD are to reduce cost 
and minimize flow times in the design and factory production 
of electrical hardware. This calls for identification of areas 
in the equipment design, packaging, test and fabrication 
processes that are operationally and economically adaptable 
to computerized techniques and the specification and de¬ 
velopment of those techniques. 

SYSTEM DESCRIPTION 

The following is a brief description of the Electronic Com¬ 
puter Aided Design (E/CAD) system in use at The Boeing 
Company. 

EICAD system description 

Some of the factors making the development of electronics 
systems a lengthy and time-consuming process relate to the 
sheer mass of detail work that must be performed. Examples 
are: the drafting and checking of logic diagrams, the tabu¬ 
lation of electrical parametric data (time delays, logic states, 
voltages, etc.), evaluation to verify electronics designs and 
functional test sequences, the assignment of logic elements 
to integrated circuit packages (allocation) and of integrated 
circuit packages to printed circuit boards (partitioning), 
placement of circuit packages on the boards, boards in card 
cages or drawers, and the production of documentation to 


convey fabrication details to the shop. This accumulation of 
data during development of a product is an essential pro¬ 
cedure, from preliminary design analysis through manufac¬ 
turing. 

To facilitate efficient transfer of data, the E/CAD system 
was designed to ensure that the programs do not commu¬ 
nicate directly with each other but through common groups 
of data residing in the design data base. This facilitates the 
addition of new software programs to the system. A separate 
Data Base Utility program is provided to allow the user to 
add user-prepared data to the data base and to store and 
control data. 

The Boeing E/CAD System is schematically represented 
in Figure 1. The rectangular boxes represent programs that 
are operational on Company computers, while the oval- 
ended boxes represent manual design functions and are 
entry points into the system. The programs are divided into 
two categories: design validation (including analysis, simu¬ 
lation and functional test) and packaging. Analog analysis, 
digital analysis, Computer Simulator, logic simulator, and 
digital functional test comprise the design validation pro¬ 
grams. They assist the user in converting the data repre¬ 
sented by a schematic or logic diagram into a format that 
can be used by the E/CAD System and provide aids to 
detect and correct logic errors in the system design. The 
Logic Simulator program assists the engineer in performing 
detailed logic simulation (design verification) of digital as¬ 
semblies, including automatic worst case timing analysis. 
The Digital Functional Test program aids in the efficient 
development of high quality functional tests. Also included 
are post-processing programs which translate the test output 
data to releasable documentation and format the data for 
use on automatic test equipment. In addition, numerous 
analog circuit analysis programs (e.g., ISPICE, CIRCUS, 
SCEPTRE) and thermal analysis programs (e.g., BETA) are 
utilized but are not currently integrated into the E/CAD 
system. 

The Packaging Interface program is the interface between 
design validation and packaging design. User-supplied pack¬ 
aging information (the assignment of logic devices to parts 
and parts to assemblies) is combined with the design de¬ 
scription from the analysis programs. The Packaging Inter- 
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Figure 1—E/CAD system diagram 


face program ensures against incompatibilities between logic 
design and packaging design information, determines all sig¬ 
nals that must be brought to the connectors on the assembly, 
automatically assigns pin numbers for all logic devices, and 
generates pin-node data for use by the packaging programs 
and the functional test patterns for producing the final test 
program package. 

The packaging programs are divided into two major cat¬ 
egories: printed circuit board design and backplane design. 
Design of a printed circuit board (PCB) is accomplished with 
a combination of large-scale computer and small midicom¬ 
puter with interactive graphics. The PCB editor utilizes 
packaging design information in addition to the data from 
the Packaging Interface program. Packaging design infor¬ 
mation (design rules, pad sizes, spacing, optional part 
placement, etc) is entered into the system, processed, and 
edited to correct errors. The resulting information is used 
for interactive placement of components and automatic rout¬ 
ing of conductors. Once automatic routing is completed, 


options are provided for manual intervention and modifica¬ 
tion of conductor routes by means of the interactive graphics 
console (Figure 2). Automatic checking for connection and 
dimensional spacing violations is a part of the software sys¬ 
tem. Once board layout is completed, necessary information 
is generated to drive plotters to produce assembly drawing 
and photo master artwork and a control tape for numerically 
controlled drilling is produced. The assembly drawing, photo 
master artwork, and drill tape, when combined with other 
manually prepared instructions, are used to produce the 
finished printed circuit board. 

The second major category of packaging programs is 
backplane design, which involves point-to-point wiring pre¬ 
dominantly by wire wrap. This packaging technique is used 
for two general categories of assembly design: digital logic 
assemblies and PCB connector backplanes. Wire wrap as¬ 
sembly of digital logic assemblies usually proves to be more 
cost effective than PCB’s in applications such as bread¬ 
boards and small-quantity production runs. The shop aids 
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Figure 2—Interactive PCB design 


necessary to wire the assembly are produced from the wiring 
program in the form of a punched tape and a computer 
listing. This tape and listing are used by the shop for fabri¬ 
cation. The wiring program also places data on the data 
base for use by a documentation program to produce a 
complete wire book. The wire book contains necessary in¬ 
formation to identify the wiring on an assembly and maintain 
unit configuration control. The documentation program, in 
turn, places data on the data base for use by the DITMCO 
Test-Postprocessor program, which generates a test tape for 
the DITMCO test equipment to check for open or shorted 
circuits in the completed wiring assembly. 


Utilization within boeing 

The E/CAD system has been in development since late 
1970. An overall system approach was selected and capa¬ 
bilities added on a modular basis. The E/CAD system is 
used on all major electronic design and development activ¬ 
ities within The Boeing Company. To date, approximately 
2,000 PCB’s and 700 wired assemblies have been designed 
using these capabilities. The average PCB is four to six 
layers and the average wired assembly has in excess of 1,000 
wires. Cost savings/avoidance to date are in excess of 10 
million dollars. 
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Home & hobbying computing 


OVERVIEW 

Computers became consumer products around the beginning of 1975. In less 
than three and one half years, a considerable array of companies have come into 
existence to address the new marketplace of “personal computing.” This portion 
of the Conference Program addresses the topic of home and hobby computing 
from several aspects. 

A survey is provided of the brief “history,” the current situation, and the 
foreseeable futures of personal computing. Following this overview, a discussion 
will address the variety of usual and unusual hardware, software, and systems 
that have appeared in the personal computing application areas. This will include 
comments regarding the problems and promise of home computing hardware, 
software, and applications. 

To date, personal computing has been limited to the fifty to one hundred 
thousand computer enthusiasts who currently own home computers. This is not 
a mass consumer market, however it is evident that such a market is about to 
open up, in some ways comparable to the market for stereos, television, and CB 
radio. The final session in this Technical Area of the Conference Program will 
address this topic—providing computer power for the average lay person; the 
mass consumer who will use computers only to the extent that they may be used 
without expertise, and only to the extent that they can solve problems that the 
average person may be interested in solving. 


THEMES 

SESSION 1: Computers became consumer products around the beginning of 
1975. Since then, a multitude of companies have been created to address the new 


357 


358 


National Computer Conference, 1978 


marketplace of “personal computing.” There are 50,000-90,000 computers in 
people’s homes, today. The speaker will briefly survey the “history” of this 
movement, detail the present situation, and hypothesize concerning the foresee¬ 
able future. Panelists will address various aspects of the alternative foreseeable 
futures. 

SESSION 2: A variety of usual and unusual hardware, software, and applica¬ 
tions are appearing in the world of computers as computers become personal 
devices. These new facilities provide a wide range of problems and promise. The 
speaker will provide an overview of the positive and negative aspects of personal 
computing equipment, programs, and uses. Panelists will present a variety of 
views regarding the hard and soft facilities that are unique to consumer comput¬ 
ing. 

SESSION 3: The microcomputer emerged from a laboratory curiosity to a 
hobbyist toy and from there to a usable business and industrial tool. What of its 
potential as a mass merchandised electronics device (like radios and televisions) 
which can be used by persons technically disinterested and incompetent to do 
other than turn a knob or two? 

Representatives from companies which are near the leading edge of this market 
will present their views and insights in this session. 




Personal Computing—A little past 
and a lot of future 


by PORTIA ISAACSON 

The Micro Store 
Richardson, Texas 

INTRODUCTION 

We are incredibly lucky! When I look back in time, I see no 
time more exciting than now! When I look forward in time, 
I see no time more exciting than now! For now, we are 
privileged to witness the dawn of the era of personal infor¬ 
mation processing. Few people have ever lived at such a 
time of technological change. Perhaps only those who were 
present when the first book was printed have, before us, 
had a similar opportunity to observe the birth of a technol¬ 
ogy-derived tool having the power of the personal computer 
to improve our quality of life. We are presently witnessing 
what is probably the most significant of all technology-based 
revolutions surpassing in importance the printing press, the 
automobile and the assembly line. 

Low-cost abundant computing brings us to the dawn of a 
new era. A new era in which information processing power 
will no longer be the exclusive tool of government and large 
business. Rather we will have computers for people to use 
in a near limitless variety pf ways in our work, our play, 
and all aspects of our daily lives. 


THE COMPUTER HOBBY 

Five years ago, and even before, those of us involved 
with microprocessors were making rather accurate predic¬ 
tions as to the declining cost of computing resources based 
on our knowledge of the new microprocessor technology. 
We did not predict, however, the absolutely incredible 
growth of hobby computing. 

Although there had been some activity previously by 
really enterprising do-it-yourselfers, the computer hobby 
really took off in January, 1975, when the MITS Altair 
Computer appeared on the cover of Popular Electronics 
magazine. 1 The computer was available in kit form for about 
$400. A new hobby and a new industry were bom. 

In July, 1975, Dick Heiser opened the first retail computer 
store. Many others were to follow. The computer store 
started making its place in the computer industry as a new 
way to market and service low-cost computers and related 
equipment. 


In the summer of ‘75, MITS was driving its van all over 
the country demonstrating the Altair computer. In city after 
city computer hobby clubs were bom at the MITS van, 
which brought the computer hobbyists in each geographic 
area together for the first time. Now computer hobby clubs 
number more than 300, with several in other countries. One 
of these clubs has membership numbering in the thousands 
and hundreds is very common. Soon these clubs will join 
together to form an international amateur computer society. 

What sort of person is this computer hobbyist? He or she 
is a hardware or software tinkerer—an experimenter. The 
computer hobbyist comes complete with clubs, conventions, 
magazines, T-shirts, and bumper stickers. Hobby computing 
will surely become the elite of the technological hobbies. 
Who is this computer hobbyist? He or she is an engineer, 
a programmer, an electronics buff, an amateur radio enthu¬ 
siast, a lawyer, a small business person. Most actually have 
surprisingly serious intended uses for their computers. In 
fact, a surprising number are depreciating their computers 
on their income tax returns. 

Personal computing is not only hobby computing. The 
fact that hobby computing has received so much publicity 
is really misleading. The graph in Figure 1 illustrates this 
point. Discussions with several computer stores indicate 
that the personal computing equipment sold over the past 
year has gone, for the most part, to business, industry and 
educational institutions. Only about 20 percent of the per¬ 
sonal computers sold this past year have been sold to hob¬ 
byists. To further illustrate this point, customers of my com¬ 
puter store include: oil companies, universities, hospitals, 
trucking companies, security companies, public schools, 
personnel agencies, power and light companies, manufac¬ 
turing companies, and electronics companies. 

Personal computers are replacing time-sharing services in 
businesses large and small. Personal computers are replacing 
mini computers in process control applications. Addition¬ 
ally, personal computers are finding their way into new 
applications where the new low cost of computing makes 
possible the use of computing where before it was too ex¬ 
pensive. One way to view personal computing is that it is 
simply low-cost computing. And this new low-cost comput¬ 
ing will compete with all other traditional forms of comput¬ 
ing resources. 
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A NEW INDUSTRY 

A new industry has emerged to support the personal com¬ 
puting market. There are more than 100 companies manu¬ 
facturing personal computing products. There are more than 
500 computer stores selling and servicing these products. 
There are 10 major magazines dedicated to personal com¬ 
puting. 

The new computer industry differs in several important 
respects from the computer industry that preceded it. 
These differences include fierce competition around a stand¬ 
ard internal bus and a new way of marketing and servicing 
computers. 

The standard bus 

Fundamental to this difference and to the development of 
the personal computing industry itself has been the S-100 
bus standard. The S-100 bus is a set of electrical and logical 
specifications for the connections between the various 
printed circuit cards that may share the bus. These printed 
circuit cards can include: a CPU card, memory cards, key¬ 
board and video-display interface cards, audio cassette in¬ 
terface cards, floppy disk interface cards, music synthesis 
cards, voice recognition cards, joy stick control cards, and 
so on. The S-100 bus was originally the MITS Altair bus. 
Now there are at least twenty companies manufacturing S- 
100 bus computers. There are more than 150 different prod¬ 
ucts including computers, memories, disks, printers, graph¬ 
ics displays, joy sticks, music synthesis equipment, voice 


recognition boxes, voice generation boxes, and home elec¬ 
trical system controls presently being manufactured for the 
S-100 bus. The broad range of available S-100 bus equipment 
greatly exceeds the offerings of any one manufacturer. 

The most important aspect of the S-100 bus standard has 
been its impact on the evolution of the personal computing 
industry. Up to this time most of the companies manufac¬ 
turing personal computing products have been small com¬ 
panies, including many garage-type operations. Few of these 
companies could afford to manufacture and market an entire 
computer system and most of them offer only one or two 
types of boards. The S-100 bus standard has created a mar¬ 
ketplace large enough to enable a small company to be 
successful in the market on a low budget offering only a 
single board that provides some nice option for a S-100 bus 
computer. 

The small entry cost to the S-100 bus market has made 
the competition fierce. If a product can be manufactured 
and marketed for less, it will be. Products are priced very 
close to the manufacturing cost. This is quite unlike the 
traditional computer industry which is well-known for prices 
that have little relation to the manufacturing cost but rather 
have been priced by looking at what the market will bear. 

The computer store would not have been nearly so viable 
a concept without the standard bus. Virtually all the personal 
computing manufacturers have been under capitalized, re¬ 
sulting in long waits for products. In many cases stores have 
been forced to buy essentially the same product from several 
different manufacturers in order to be assured of an adequate 
and timely supply. Additionally, since many of the products 
in a computer store are interchangeable, inventories of par¬ 
ticular brands can be reduced. A store can choose a product 
such as an S-100 bus memory board from among a large 
number of possible suppliers, using consideration such as 
delivery time and margin as primary parameters rather than 
being forced to take whatever the CPU manufacturer cares 
to offer. This kind of competition has enabled stores to get 
nearer the kind of margin and service they need. 

The retail computer store 

The new low-cost computer products require a new way 
of marketing and servicing these products. It no longer 
makes sense for a salesperson or field engineer to call on 
the individual customer when his computing system costs 
only a few thousand dollars. The retail computer store fills 
this need. Computer stores offer a place to test drive several 
computing systems before buying one. Stores try to offer 
off-the-shelf delivery of computers and associated equip¬ 
ment and software. Computer stores offer advice on config¬ 
uring a system for a particular application from the wide 
variety of personal computing products available. Most 
stores repair the equipment that they sell. Some stores offer 
hardware and software consulting. Most offer a wide as¬ 
sortment of computer books and magazines. Some stores 
hold classes in using and programming personal computers. 
The computer store plays a new and essential role in the 
new computer industry. 
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THE HOME COMPUTER REVOLUTION 

In late 1977 we saw the first signs of the real home com¬ 
puter revolution. The fully assembled complete computer 
was offered by two different companies, Radio Shack and 
Commodore, for under $600. These computers include the 
CPU, memory, a keyboard, video display, and read-only- 
memory Basic language interpreter. So now we have the 
hardware in the right price range for the massive home 
market, but still missing are the huge libraries of programs 
that will make these computers really useful. This software 
will come. Eventually we’ll buy programs like we buy stereo 
records—from a wide selection—where some program can 
be found to satisfy nearly any taste or requirement. When 
these program libraries are available, millions of home com¬ 
puters will be sold. 

Now what would motivate millions of people to buy home 
computers? The two primary reasons are entertainment and 
education. 

Do not underestimate the sales appeal of the computer 
game as an entertainment form. Look carefully at the com¬ 
puter as a new form of personally available, intellectually 
challenging, interactive entertainment. Of course, the com¬ 
puter offers simple games like pong, target, and guess-the- 
number. Much more importantly, the computer offers the 
opportunity for simulated experiences without risk. For ex¬ 
ample, the well-known Star Trek game provides the player 
the experience of being a star ship captain—the opportunity 
to make monumental decisions on which rests the fate of 
the universe. Other games let us pretend to be a ruler of a 
country or the president of a company. The simulated ex¬ 
perience-type of computer game is a powerful new enter¬ 
tainment form. A $600 price tag seems very small for just 
this one function of the home computer. 

Education is the other major reason computers will be sold 
to millions. The home computer coupled with well-known 
computer assisted instruction techniques will offer an inter¬ 
active personalized form of instruction for the home. CAI 
programs will be available for instruction in subjects as far 
ranging as arithmetic drills, French grammar, and driver 
education. This new home educational medium may even¬ 
tually have a dramatic impact on our educational systems. 
When the computer-assisted home education center has 
evolved to its full potential using such technologies as video 
disks and two-way electronic communication links to 
schools, the home could very well become the center for 
educational acitvity. The school’s function would be to pre¬ 
pare instructional materials and certify levels of achieve¬ 
ment. 

Entertainment and education will surely be reasons 
enough for many people to buy home computers. Of course, 
the home computer will have many other uses including: 
personal financial management, astrological forecasting, 
keeping inventories of collections such as stamp or beer can 
collections, centralized control of home physical systems, 
maintaining address lists, and use as an artistic medium in 
graphics and music. In fact, the uses of computers are as 
varied as the individuals who apply them. Applications of 


the home computer have little limit other than the imagina¬ 
tion. 


COMPUTER INDUSTRY CHANGES 

The new low-cost computing will cause many changes in 
the traditional computer industry. Some of the changes are 
starting now. Some will not reach their full impact for years 
to come. The very structure of the data processing organi¬ 
zation will be impacted. Additionally, the individual data 
processing professional may find his or her role changing 
significantly. 

Personal computers are replacing and are competing with 
mini computers in low-end applications including process 
control and small business. In order to understand why 
personal computers are able to successfully compete with 
minis in many applications, it must be understood that per¬ 
sonal computers are not small—they are just inexpensive. 
A recent article compared the performance of an IBM S/360 
Mod 30, the work-horse business data processing computer 
of the late 1960’s, to the personal computer. 2 According to 
some parameters the personal computer actually exceeds 
the S/360 Mod 30 in computing capability. 

As personal computers move into the mini computer low- 
end market, minis will be forced to look upward for more 
applications. They have already started eyeing the strong¬ 
holds of the large mainframes. 

Personal computers are replacing time-sharing services. 
Time-sharing was devised as a means of bringing computing 
resources to the individual. Now that the individual can own 
an entire computer, there is simply little need to time-share 
a large computer. The advantages of a 16-user personal 
computer laboratory over a 16-user time-sharing system 
were recently documented. 3 They include the fact that the 
personal computers are half the price of the time-sharing 
system, the personal computing laboratory can be built in¬ 
crementally with a very low entry cost, personal computers 
offer the latest in technology, the personal computing lab¬ 
oratory is more reliable and available more of the time, and 
the individual can typically have more disk storage space. 
Disadvantages of the personal computing laboratory include 
the lack of large software libraries and the fact that very 
large programs that might use the entire time-sharing com¬ 
puter during non-time-sharing hours cannot be run. In many 
cases the personal computer offers a better way than time¬ 
sharing of bringing computing resources to the individual. 

The data processing center will soon find itself loosing 
control of the corporate data processing function. This has 
already happened to a certain extent as mini computers first 
meant for process control applications were found useful as 
general purpose data processing machines. Departments 
within corporations found that by buying their own small 
computer they could spend significantly less on data proc¬ 
essing than if they bought computing services from the data 
processing center. This trend will be greatly accelerated by 
the still much lower cost of the personal computer. The 
eventual result of this type of pressure on the centralized 
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data processing facility will be pressure on the large main¬ 
frame manufacturer to lower prices to compete with the low- 
cost computers. Ultimately personal computers will force 
lowering of prices by the large mainframe manufacturers. 

A major change for the computer professional will be the 
fact that the public will be computer literate. The public will 
have a much better understanding of computers and soft¬ 
ware. No longer will they be awestricken and unquestioning 
about a computerized process. They will understand the role 
of the programmer and demand that programs interfacing 
with the public offer reasonable human interfaces and con¬ 
sideration. 

Today society’s view of the computer programmer is with 
considerable awe and respect. Other departments in a cor¬ 
poration view the data processing person as the practitioner 
of the black art of computer magic. Most of us in computing 
have at one time or another used our technological jargon 
to our advantage in cutting short a discussion. All this will 
change when computing and programming become house¬ 
hold activities. The computing profession will not be viewed 
with quite the splendor it once was. This author occasionally 
reflects on the fact that a Ph.D. in Computer Science will 
have the same social prestige as a Ph.D. in refrigerator 
science when the computer becomes just another household 
appliance. 

It is well-known that one of the biggest problems in the 
computer industry today is the high cost of software. Low- 
cost computing may eventually help solve that problem in 
an indirect way. Low-cost computing will lead to widely 
available computer education at all levels. Now that the cost 
of a computer is not much different than the cost of a 
typewriter, the major block to widely available computer 
education is gone. With computer education abundant there 
will be a flood of qualified entry level programmers. The 
effect may ultimately be more competition for data process¬ 
ing positions and ultimately lower salaries for programmers. 
The result: a decline in software costs. A possible weakness 
in this scenario is that at the same time the number of com¬ 
puter trained people is increasing, the amount of software 
needed by new low-cost computing applications is also in¬ 
creasing. Perhaps the need for programmers will be adequate 
to absorb the higher production. 

CHANGES IN OUR WAY OF LIFE 

Predicting the future is always a risky venture at best. 
However, we can take a few peeks at what the effects of 
abundant low-cost computing may be. We have already dis¬ 
cussed the fact that to our homes the computer will bring 
powerful and exciting new forms of entertainment and ed¬ 
ucation. 

For those skeptical about the magnitude of the potential 
impacts of the home computer a simple scenario usually re¬ 
moves any doubts. What if a million people, or a thousand, 


or even just a hundred all owned the same type of computer; 
all used the same stock market analysis program, and— 
based on predictions of the program—all made the same 
moves in the market? Chaos! 

Probably the largest contributing factor to change is that 
now computer-inventiveness is in the public domain. No 
longer can only large business and well-endowed universities 
invent devices having computers in them, computers them¬ 
selves or computer-related equipment. Now because of the 
new low cost of computing, the average person can use 
computers as components in inventions giving them a new 
dimension—intelligence. History tells us that many of our 
most personally useful inventions were not invented by large 
corporations or universities, but rather by an individual with 
a problem to solve. Now that same kind of motivation and 
inventiveness will be applied to computer products. In fact, 
from the present computer hobby movement will come giz¬ 
mos too numerous to mention. We are now on the threshold 
of a gizmo age in which we will be surrounded by intelligent 
devices supplying services that would be impossible to imag¬ 
ine at this time. 

All is not good, however. For example, it is now possible 
to build a computer-based device that will place telephone 
calls, play a recorded message, record the verbal response, 
and even accept the touch-tone input of a credit card num¬ 
ber. This device can be very persistent and devious in plac¬ 
ing calls. For example, it can either place calls against a 
provided list of numbers or it can generate the numbers 
within a given prefix which, of course, corresponds to a 
geographic area. Notice your unlisted telephone number is 
of no help. The call-placing device can remember that you 
didn’t answer and place the call periodically until you do. 
It could even remember that you hung up! The devastating 
truth is that a device can be built to place junk telephone 
calls for about per call compared to per letter in 
postage alone for junk mail. The only answer is probably 
legislation. In the meantime a low-cost device can be built 
that will answer your telephone and require entry of a touch- 
tone password before the household is notified of the call. 

Over the long range the abundant computer will add a 
new and rewarding dimension to our lives. We have a new 
fundamental tool, a new medium for expression and expe¬ 
rience. We are indeed incredibly lucky to be present at the 
dawn of the era of personal information processing. 


REFERENCES 

1. Warren, Jim C., Mark E. Deppe, and James P. Fry, “Personal Comput¬ 
ing—An Overview for Computer Professionals,” AFIPS Conference Pro¬ 
ceedings, Vol. 46, 1977, pp. 493-498. 

2. Isaacson, Portia, Datamation, Personal Computing Department, Novem¬ 
ber, 1977. 

3. Isaacson, Portia, Proceedings of the 1977 Computer Users Conference, 
Computer Science Department, East Texas State University, Commerce, 
Texas, March 25, 1977, pp. 59-68. 




Personal computers—Hardware, software and 
documentation 


by JEF RASKIN 

Brisbane, California 


INTRODUCTION 

What is a personal computer? Possible candidates for per¬ 
sonal computers range from programmable calculators and 
games on one end to an antique IBM 7094 in a hobbyist’s 
basement on the other. There is no doubt that the pocket 
sized programmable HP-65 and its successors are computers 
in the classical sense. They can be programmed to execute 
any algorithms that fit within their memory limitations. This 
is all that can be said of any computer. For this discussion, 
a useful dividing line between calculators and computers is 
that calculators do not provide for the input and output of 
natural language text. 

It is more difficult to draw a line between personal com¬ 
puters and other devices that are correctly called computers. 
In fact, there is no special term for those computers not 
intended to be personal computers. The term “impersonal 
computers” is tempting. For this discussion (and at this 
point in history) personal computers will be defined as those 
computers that are marketed as personal computers. This 
correctly reflects the fact that there is nothing special about 
personal computers except that they are inexpensive enough 
to be reasonably sold to large numbers of individuals. 

Excluded from this discussion are truly home-brewed 
computers, designed and built by a rugged individual for his 
or her own edification and amusement. Also excluded from 
this discussion are computers, however small, marketed as 
small business systems. They may, in fact, be identical to 
the systems that are being discussed, but very different 
considerations apply to their software and documentation— 
they are also directed toward a different audience. Business 
users will apply different criteria than home users to the 
selection of a system. During business hours, at least, they 
are not personal computer systems. It is less than reckless 
to predict that they will have their own publications and 
conferences in the near future. 

The systems considered here are those that are being 
programmed and used by thousands of people, each of 
whom can point to the computer and say, “It’s mine.” 

HARDWARE 

A personal computer is based on one of the commercially 
available microprocessors. Most begin with an 8080 or a 


6800, or their successors (such as the 8085, 280 or the 6500 
series). The hardware achieved commercial quality very 
quickly, except for a few design errors, some of which have 
become almost legendary. By and large, personal computers 
are reliable. They use the same quality components, boards 
connectors, enclosures, and manufacturing processes as 
commercial computer products—with only a few brands cut¬ 
ting comers. 

PERPHERALS 

Anybody familiar with minicomputers might expect that 
paper tape readers and punches would be major peripherals 
in micros. It is not so. Cassette tape machines and ordinary 
commercial audio recorders dominate the field. Simple but 
clever interfaces make them moderately reliable off-line 
storage devices. There used to be a certain fascination in 
buying your computer peripherals on special at a discount 
store. Since you will soon be able to buy the whole computer 
system there, the novelty will quickly wear off. 

One of ten best personal computer peripherals is the mem¬ 
ory-mapped video display. The speed and capabilities of 
these displays—which drive an ordinary TV monitor—rival 
and often best even the more expensive “intelligent” ter¬ 
minals. Their main limitation is their small (typically 16 line 
by 64 character) display. Many TV drivers display fewer 
characters than that. Nonetheless, most mm/-manufacturers 
have far poorer console displays. Most of the personal com¬ 
puter displays provide some form of graphics as a standard 
feature; a few brands even offer color graphics. 

Floppy disk drives and controllers do not differ signifi¬ 
cantly from the rest of the industry. Hard disks for personal 
computers are not yet available, and may never be; it all 
depends on the price of various new forms of memory. 
Again, the personal computers are in the same position as 
the rest of the industry. Keyboards are mostly encoded, 
with a few more advanced systems going to polled operation. 
Keyboards are rapidly being integrated into products. 

Paper tape is used. The only popular punch is the ASR33. 
Some clever tape readers have appeared, including at least 
two that can go as fast as you can pull the tape through. 
Since they work asynchronously, all one needs is a simple 
electric motor and a spool to have a cheap and remarkably 
fast tape reader. 
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Punched cards have no hold in the personal computer 
industry at all. 

The number and variety of peripherals and interfaces for 
personal computers are incredible. Just one system the pop¬ 
ular “S-100” bus, has over 300 different devices that are 
mechanically and electrically (more or less) compatible. 
Never has one bus been available on so many different 
computer brands. Never have so many hundreds of com¬ 
panies made accessories for the same bus. 

MEMORY 

It took the personal computers to show the industry how 
cheaply memory could be sold. The personal computer de¬ 
signers have not been shy at using the latest memory chips 
as they have become available. In hardware, the personal 
computer user has the latest, the best, and the cheapest. 

SOFTWARE 

Personal computer software, for the most part, does not 
attain the state-of-the-commercial-art level that the hard¬ 
ware does. There is a unique problem in the hobbyist indus¬ 
try: almost everybody has a choice of only a few processors. 
In the mini and maxi computer world, manufacturers could 
afford to develop software, since it could only be used on 
their brand of computer. This problem has only begun to 
creep into the mini and maxi computer world. At present, 
it is not clear that a substantial investment in software de¬ 
velopment can return a profit. Selling packages to individ¬ 
uals at $5.00 to $50.00 a shot is working for some small, 
low-overhead organizations and individual free-lance pro¬ 
grammers. Writing the obligatory BASIC interpreter for var¬ 
ious companies is keeping more than one firm alive. 

Very few owners of personal computers are aware of the 
benefits of the more advanced computer languages, and 


therefore, are content with an endless stream of ad hoc 
additions to BASIC. Since the structure of BASIC makes 
writing long and complex programs lugubrious, programs 
tend to be short. Since the overhead for remarks is high, 
they tend to be poorly documented at best. 

One of the most widely used assemblers for the 8080— 
you may not believe this—is missing a few instructions! 
Assemblers that offer macros and relocatable code are only 
beginning to be available. Noise about APL is often heard, 
but the product lags far behind. FORTRAN is beginning to 
be used, and there is some interest, for some reason, in 
COBOL. FORTH is available but overpriced for most of the 
market. 

An amazing software development has been the versions 
of Tiny BASIC. They use interpreters whose object code 
can easily be displayed on a single quarto-sized page. The 
cheap personal computer has forced such development on 
one hand, and made it usable on the other. The disk-op¬ 
erating systems are rapidly gaining in sophistication. Per¬ 
sonal computer software is a funny mixture of cleverness 
and conservative copying of outmoded programming tech¬ 
niques. 


DOCUMENTATION 

Personal computers are being marketed mainly at first¬ 
time users. That is not the current user population—that is 
the anticipated user population. With few, if any, exceptions 
there aren’t any computers systems that you can hand a 
beginner, then come back a week later and find the person 
happy and the computer up and running. The fault is partly 
with the software and mostly with the documentation. This 
will have to change radically for the better in the next few 
years, or personal computers will continue to be the exclu¬ 
sive toys of the electronics engineers and programmers who 
now buy them. 
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SIMULATION 


An increasingly important application of computers is their use for simulation. 
Simulation involves performing experiments on models instead of the real-world 
system—the simuland—for obvious reasons. Simulation is cost-effective whereas 
experimenting with real systems is usually more expensive, may be dangerous, 
and is often impossible because the simuland is not available, or is intractible to 
experimental manipulation. 

In recognition of the current and exponentially increasing importance of the 
use of computers for modeling and simulation the NCC Program Committee has 
dedicated a Program Area to the subject. 

There are three sessions in this Simulation Area. One is a panel discussion 
during which panelists will address the question “Can Large-Scale Models Al¬ 
leviate World Resource Problems?” The written Position Statements of each 
panelist has been distributed to all other panelists with the expectation that each 
will thus be prepared better to discuss points raised by the others. However, 
besides setting the stage for a more enlightening discussion, this was done because 
panelists were selected to some extent because of their differing points of view 
and it was thought “forewarned would be forearmed” with pertinent material. 

An effort was made to keep the number of panelists relatively small so that 
each could have a reasonable time in which to present his views, then defend 
them if they are challenged either by other panelists or by members of the 
audience. On the other hand the panel organizer and moderator has invited 
knowledgeable (and sometimes opinionated) simulationists of his acquaintance to 
join in the discussion from the audience. 

In addition to the panel session there are two simulation sessions for the 
presentation of technical papers. The subject of one of these is: 


MICROPROGRAMMING AND SIMULATION 

This session was arranged by Dr. Lance A. Leventhal, Society for Computer 
Simulation, in cooperation with Dr. Gary Nutt, Department of Computer Science, 
University of Colorado, Boulder, Colorado, the Session Chairman. 
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The concept for this session was developed because new technical develop¬ 
ments have extended the range and power of simulation. One long-standing 
problem however, has been the development of software and systems for com¬ 
puters which are unavailable for use, unsuited for development, or not yet 
even built. Simulation has long been used to alleviate this problem while allowing 
experimentation with new architectures. Recent hardware developments have 
made it possible to create special computer facilities for these purposes. Such 
facilities can serve as universal hosts, capable of simulating (or emulating at the 
microprogram level) a wide variety of computers. One paper in the session on 
“Microprogamming and Simulation” presents an overview of the uses and ad¬ 
vantages of emulation. A second paper describes an emulation facility while a 
third paper discusses an example application. 

These papers are: 

“Emulation: Tool for Software Development,” by N. F. Schneidewind, De¬ 
partment of Computer Science, Naval Postgraduate School, Monterey, CA. 
93940; 

“PRIM System: a Framework for Emulation-based Debugging Tools,” by J. 
Goldberg, A. Cooperband, and L. Gallenson, University of Southern California 
Information Sciences Institute, Marina Del Rey, CA. 90291, and 

“A Microprogrammed AN/UYK-20 (V) Emulation,” by D. A. Deel and W. A. 
Burkhard, Computer Science Division, Department of Applied Physics and In¬ 
formation Science, University of California, San Diego, La Jolla, CA. 92093. 

The subject of another session is: 


PERIPHERAL AND MULTI-PROCESSORS IN SIMULATION 

Also arranged by Dr. Lance Leventhal, Society for Computer Simulation, this 
one with the cooperation of Dr. Walter J. Karplus, Computer Science Depart¬ 
ment, University of California, Los Angeles, Session Chairman. 

The fact that hardware can now be inexpensively produced in configuration 
specifically suited to simulation applications inspired this session. This is impor¬ 
tant because most simulation problems involve a large number of concurrent 
activities and hardware elements can be connected in parallel to handle such 
problems. Several different approaches are possible; the parallel system may be 
an add-on to a general-purpose computer, a general parallel multiprocessor may 
be built, or a parallel system may be specifically configured for a particular class 
of problems. One paper in the session on “Peripheral and Multi-processors in 
Simulation” describes a system that can be used with many large computers. 
Two other papers describe the use of multiple processors working in parallel to 
solve plasma and fluid flow simulation problems. 

The papers in this session are: 

“A Multiprocessor Digital Computer for Dynamic Systems Simulation” by R. 
M. Howe, Dept, of Aerospace Engineering, University of Michigan, Ann Arbor, 
Mich, and E. Gilbert, Applied Dynamics International, Ann Arbor, Mich. 48104; 

“Plasma Simulation on the UCLA CHI Computer System” by R. W. Huff and 
C. C. Wu, Dept, of Physics, University of California, Los Angeles, Los Angeles, 
CA. 90024; and 

“A Multiple Microcomputer Approach to Fluid Flow Problems” by J. Stein- 
hoff, Research Department, Grumman Aerospace Corporation, Bethpage, N.Y. 
11714. 




Emulation—Tool for software development 


by N. F. SCHNEIDEWIND 

Naval Postgraduate School 
Monterey, California 


INTRODUCTION 

A major computer operations problem is the conversion of 
programs from one language to another when a replacement 
computer is acquired. Emulation was developed as one so¬ 
lution to the conversion problem. Emulation allows the ma¬ 
chine instructions of the emulated (target) machine to be 
executed on the emulating (host) machine. Thus permanent 
program conversion is avoided. Frequently, emulated pro¬ 
grams execute several times faster than their non-emulated 
counterparts. This occurs because the host is faster than the 
target or its microinstructions are more efficient than the 
target’s machine instructions, or for both reasons. Other 
methods of program conversion which compete with emu¬ 
lation are: recompilation, reassembly, translation and sim¬ 
ulation (interpretation). 

Although “translate” is frequently used as a general term 
to denote all types of language conversion, including assem¬ 
blers and compilers, here it will be used more specifically 
to mean a conversion from language A to language B where 
A and B are not machine languages. It should be noted that 
recompilation, reassembly and translation effect a perma¬ 
nent conversion, i.e., the result of the process is a program 
in another language, whereas emulation and simulation em¬ 
ploy another language, microinstructions and interpretive 
routines, respectively, only during the period of program 
execution. Another way of viewing this difference is that 
execution of a converted program using recompilation, reas¬ 
sembly or translation is a two step process-conversion fol¬ 
lowed by execution—whereas emulation and simulation are 
one step processes; as far as the user can discern, conver¬ 
sion and execution take place simultaneously in a single 
computer process. 

Another application of program conversion techniques is 
in a software development organization, where a multiplicity 
of hardware and software systems are produced. This oc¬ 
curs, for example, in organizations which are developers of 
tactical software systems for ships and aircraft. Typically, 
many computers comprise a single system. The variety of 
computers is even greater across the many systems with 
which the software development organization could become 
involved. It would be quite expensive for the software or¬ 
ganization to acquire all the required computers and pro¬ 
gramming languages, particularly in view of the transient 


nature of the development process. In addition, the com¬ 
puters and languages which are needed for testing purposes 
are frequently unavailable at test time. If the organization 
were to wait until all hardware and software is designed and 
implemented, system integration would be delayed to the 
point of causing delivery slippage. Thus a procedure is nec¬ 
essary for executing and testing programs prior to the avail¬ 
ability of the hardware which will be used in the delivered 
system. In addition to the hardware being unavailable, it is 
possible that compilers to be used with the hardware are 
also unavailable at test time. This situation would require a 
method for producing programs which are not dependent on 
the use of machine language generated by the operational 
compiler. Here and subsequently, “operational” will refer 
to the software and hardware which will be used in the field 
after the system is delivered to the customer. 

This paper will describe and evaluate different alternatives 
for providing a software development organization with the 
capability of writing and testing programs prior to the avail¬ 
ability of the operational hardware and compilers. In this 
context we will describe the concept of a software produc¬ 
tion and test facility which could be used for the develop¬ 
ment and implementation of a variety of software and hard¬ 
ware systems in the absence of full operational capability. 
We will also address some of the problems which arise in 
the execution of this concept, such as the difficulty of faith¬ 
fully representing I/O operations. 

PROGRAM CONVERSION ALTERNATIVES FOR A 

SOFTWARE DEVELOPMENT ORGANIZATION 

We now explore various alternatives for achieving a mul¬ 
tihardware, multilingual capability for a software develop¬ 
ment organization. First, some terminology and notation are 
in order for the purpose of identifying the various types of 
languages and machines which may be involved in the op¬ 
eration of the software facility. The terms “target” and 
“host” are frequently used to refer to the “executed on” 
and “executed by” machines, respectively. In order to de¬ 
scribe the software development environment referred to 
above, it is necessary to expand the terminology to include 
two types of languages and two types of machines as shown 
below. It should be noted that the case described below 
differs from the situation where an organization converts 
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from system A to system B. In this case A and B represent 
the target language/machine and host language/machine, re¬ 
spectively. In the software development organization case, 
the target language/machine is the operational system and 
the host language/machine is the system used in the software 
development facility for testing and integration. 

• Source language: Language used for coding the pro¬ 

gram initially. 

• Host language: Language used for executing the 

source language program at the soft¬ 
ware development facility. 

• Host machine: Machine on which the source pro¬ 

gram is executed in the software de¬ 
velopment facility. 

• Target machine: Operational computer. 

These relationships are shown in Figures 1 and 2. Figure 
1 shows the general concept of the software development 
facility wherein two major choices exist for producing soft¬ 
ware for test and integration purposes. One method in Figure 
1 involves producing host machine language which is exe¬ 
cuted on the host machine. This method tests problem logic 
in terms of host machine language. This method could be 
implemented by selecting one of the following paths in Fig¬ 
ure 2: 

• 1, 2 Interpretation 

• 1, 3, 9, 10 Translation 

• 1, 9, 10 Compilation 

• 5, 9, 10 Assembly 

The second method shown in Figure 1 involves producing 
target machine language which is executed in the host. This 
method tests problem logic in terms of target machine lan¬ 
guage. This method could be implemented by selecting one 
of the following paths in Figure 2: 

• 1, 4, 7 or 1, 6, 7 Compilation followed by Simulation 

• 4, 7 Simulation with Assembly Language Input 

• 6, 7 Simulation with Machine Language Input 

• 4, 6, 8 Assembly followed by Emulation 

• 1, 6, 8 Compilation followed by Emulation 

• 6, 8 Emulation with Machine Language Input 

For tactical system development, the use of either method 
would be limited to testing the logic of individual programs 
and the logic of integrated programs. Tests of system timing, 
storage utilization and hardware dependent tests—perform¬ 
ance testing—would be deferred until the target system is 
available. However, a substantial portion of the testing of 
most systems involves the checking of sequence of opera¬ 
tions and the correct production of outputs for given inputs. 
The correct functioning of many of these operations is not 
speed or hardware dependent. This approach is also com¬ 
patible with the top-down method of system design which 
emphasizes the early development of the system frame¬ 
work—calling sequences and communication paths. Thus 


logic testing performed in advance of performance testing 
will facilitate the entire test process. 

In Figure 2, many alternatives are listed for the sake of 
completeness; some of the alternatives would be impractical 
for a software development organization to implement. Al¬ 
though many alternatives are shown in Figure 2, only one 
or two would actually be used in a single organization. We 
now examine each of the alternatives shown in Figure 2; 
before doing this, the languages indicated in Figure 2 will be 
identified. 

Item 1. HLL1: A host compiler exists for this high level 
language. 

Item 3. HLL2: A host compiler and translator exist for 
this high level language. A host compiler 
does not exist for HLL1. 

Item 4. AL1: Target assembly language. 

Item 5. AL2: Host assembly language. 

Item 6. ML1: Target machine language. 

Item 9. ML2: Host machine language. 

Interpretation 

• One step process. 

• Slow because of software approach used in interpreta¬ 
tion. 

• Does not execute target machine language. 

Translation 

• Two step process. 

• One hundred percent translation is not feasible. 

• Does not execute target machine language which is 
necessary for testing equipment oriented operations. 

Compilation and Assembly 

• Two step process. 

• Does not execute target machine language. 

(see Translation above) 

Simulation 

• One step process. 

• Slow because of software approach used in simulation. 
About one-half the speed of the target system. 1 

• Executes target functions but by using host machine 
instructions to simulate target machine instructions: 
modified target execution. 

Compilation followed by Emulation 

• Two step process. 

• Relatively easy to program. 

• Execute microinstructions in host which carries out 
target machine instructions. 

• Fast execution relative to simulation (at least as fast as 
target systems). 1 
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Emulation with Machine Language Input 

• One step process. 

• Difficult to program. 

• Execute microinstructions in host which carries out 
target machine instructions. 

• Fast execution relative to simulation (at least as fast as 
target system). 


EVALUATION OF EMULATION FOR A SOFTWARE 
DEVELOPMENT FACILITY 

Some of the considerations involved in deciding whether 
to employ microprogramming (emulation) for software de¬ 
velopment are the following: 

Word Length 

Is the host machine microinstruction length an integral mul¬ 
tiple or divisor of the target machine instruction? 

Program Size 

Is the program so large that target assembly or machine 
language programming would be prohibited—necessitating 
the use of a HLL and its compiler for producing machine 
language which can be emulated? 


Control Store Speed 

How fast is the control store of the host relative to its main 
memory speed? Unless the control store is several times the 
speed of the host’s native mode, the speed penalty will be 
excessive and the target machine’s timing characteristics 
become distorted. 

Type of Control Store 

Will the emulator be stored in Read Only Memory (ROM) 
or Writable Control Store (WCS)? The latter is clearly pref¬ 
erable for a software development organization. One com¬ 
puter could suffice for emulating a variety of target machines 
by changing the contents of WCS. An on-line library of 
target machine emulation programs, accessible to the WCS, 
can be envisioned. 2 Another advantage of WCS is that it 
provides a user microprogramming capability. 

Internal Data Code 

Does the internal data code in the host differ from that used 
in the target (byte versus word orientation)? Does one ma¬ 
chine use variable and the other fixed word length? 


Arithmetic Operations 

Do the arithmetic operations differ (decimal versus binary 
instructions)? 

Instruction Length and Format 

What are the differences in instruction length and format 
(size of operation code, number and length of addresses)? 

Nature of HO Operations 

Are the I/O operations of target and host vastly different, 
such as the use of programmed I/O in one and DMA in the 
other? Even a common device like a card reader may cause 
problems if the number of columns read, data code or error 
processing differ between machines. 3 Another example of 
the complexity of the microprogram which can ensue from 
attempting to emulate another common device are the con¬ 
trol signals of a typewriter terminal—carriage return, margin 
release, backspace, paper advance, etc. 4 

Microprogram Preparation Tools 

How easy will it be to prepare emulation programs? Since 
microprogramming by conventional means requires a high 
degree of programming skill, lacks good self-documentation 
procedures and is difficult to debug, research has been un¬ 
der way to develop high level languages for producing mi¬ 
crocode. 5 

Despite some of the problems of using emulation cited 
above, it can provide a powerful capability for the represen¬ 
tation of a variety of systems in a software development 
organization because of its fundamental property which al¬ 
lows the host architecture to be changed in accordance with 
different target machine requirements. 6 Rather than provide 
n sets of hardware in a software development organization, 
n ROMS can be provided or, better, n emulators can be 
recorded in a library for loading into WCS. Another alter¬ 
native is to use a combination of ROM and WCS, where the 
former is used as a backing store to hold emulator programs 
and the latter is the working storage where the current em¬ 
ulator program is stored; programs are swapped between 
ROMS and WCS. This capability would allow testing of a 
multiple computer target system. 

Examples can be cited of the successful employment of 
emulation in software development. 7 However, in order for 
emulation to be considered a general solution to the software 
development problem, its limitations must be assessed and 
circumvented. Suggestions for dealing with the limitations 
are discussed in the next section. 


LIMITATIONS OF EMULATION 

The following discussion addresses certain limitations and 
issues which are crucial to the success of an emulator ori¬ 
ented software development facility. 
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Input/output operations 

As stated previously the objective of advanced testing 
in the software development facility is limited to checking 
for correct operation and sequence of various functions; 
testing of timing relationships would be deferred until the 
target hardware, including I/O, is available. In other words, 
advanced testing checks for correctness of logic; later testing 
checks for correctness of performance. Using this approach, 
the following emulation control of I/O could be employed 
during advanced testing: Upon decoding an I/O command 
in the emulator program, control would be transferred to the 
host operating system where a timer would be set to simulate 
the duration of the I/O operation. Control would then be 
returned to the emulator. At the expiration of the timer 
interval, a timer interrupt would occur. For a read, the 
operating system would load the input buffer from the test 
data area in memory after the interrupt and prior to returning 
control to the emulator. For a write, the emulator would 
transfer data from the test data area to the output buffer; 
when the write command is decoded, control is passed to 
the operating system as described previously. In neither 
case is data transferred to or from a peripheral unit. Only 
the movement of data in memory, to or from the I/O buffers, 
is simulated. The time interval which is set in the timer 
would be drawn from a probability distribution which is 
representative of the characteristics of the target I/O device. 

Speed relationship between host and target machines 

It is important to be able to predict the speed of the target 
machine based on the speed of the host emulator. The speed 
analysis would be separated into two parts: CPU and I/O. 
In the case of the CPU operations a time ratio would be 
established based on sample measurements of time taken in 
the host to emulate specified target commands and the 
known time required for executing the target commands. 
This ratio would be applied to the number of commands to 
be executed by the target machine in order to estimate its 
CPU time. Several ratios might be needed, one for each 
type of command. The I/O timing analysis would be treated 
differently. Measurement of emulator time would not apply, 
since the emulator is not involved in I/O operations. Input/ 
output time would be estimated from target I/O equipment 
specifications and problem specifications: numbers of file 
accesses, records read/written; blocking factor, etc. 

Validation of emulator program 

There is no way to guarantee total correctness of emulator 
operation during advanced testing because all target equip¬ 
ment is not available and performance characteristics in¬ 
volving timing dependencies and storage utilization cannot 
be totally tested during this period. The best that one can 
do, as with any test, is to compare the observed operation 
with system and program specifications. In the case of ad¬ 
vanced testing, the comparison is limited to checking of: 


expected outputs for given inputs and expected sequence of 
function execution. The time of these events and efficiency 
of resource utilization cannot be evaluated. 

Figure of merit for fidelity of emulation 

It is desirable to have a simple-to-compute figure of merit 
which provides an overall measure of the closeness between 
emulator and target machine operations. A single compre¬ 
hensive measure may not be possible. If closeness is to be 
the criterion, speed ratio is not a good measure, because we 
desire the host to operate several times faster than the target 
machine. This would be advantageous for advanced testing 
because the time and cost of testing would be reduced. A 
more appropriate measure would account for host deviations 
from target operations. Examples would be the frequency 
of simulating I/O. A ratio could be computed of software 
simulated commands to total number of target commands 
processed in the host. This would provide a measure of lack 
of fidelity between emulator and target machine. 

Simulation versus emulation 

A natural competitor to emulation is simulation. Simula¬ 
tion has an advantage of being less limited in the types of 
target operations which can be performed. This is because, 
in general, software provides a flexibility for performing 
complex operations which is unattainable with microinstruc¬ 
tions. An example is the operating system task management 
and resource allocation operations which are necessary for 
executing the microprogrammed application functions. 
However, if all target functions were performed by simula¬ 
tion, there would be a lack of validity in target program 
testing because the simulator executes host machine instruc¬ 
tions. This amounts to executing in pseudo target machine 
language. In contrast, although emulated CPU instructions 
are not pure target machine language, the use of microin¬ 
structions for representing target machine control and exe¬ 
cution sequences comes closer to imitating operational con¬ 
ditions. In addition, emulators are reported to have a 
performance of five to 10 times more than simulators. 8 The 
best approach is to blend emulation and simulation so that 
the characteristics of each are used to advantage. 


CONCLUSIONS 

Alternatives for providing an advanced testing capability in 
a software development organization have been discussed. 
The need arises because some organizations are responsible 
for developing a series of systems each of which may be 
comprised of several software and hardware systems. It 
would be too expensive for the organization to acquire all 
the hardware and software systems necessary for test and 
integration. In addition many of the systems would not be 
available at test time. Emulation, combined with simulation, 
has been discussed as one solution to this problem. Its 
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advantages relative to compilation/assembly and translation 
have been examined. Although emulation is an attractive 
alternative, it should not be pursued unless the following 
conditions are satisfied: 

• An easy way of producing microprograms is needed. 
This requires a high level language. Research has dem¬ 
onstrated that it is practical to use high level languages 
for producing efficient microcode. 9 

• The host should have a writable control storage for 
allowing multiple emulator programs, representing a 
variety of target machines, to be utilized. 

• Related to the preceding items is the need for a user 
microprogramming capability. In a study of user micro- 
programmable systems it was concluded that this ca¬ 
pability can improve both the space utilization and ex¬ 
ecution time performance of a computer system. 10 

• The host should not differ greatly from the target ma¬ 
chine in terms of CPU and memory characteristics: 
word length, number of registers and memory size. 
Where differences do exist, the host should have the 
greater capability. 
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PRIM system—A framework for emulation-based debugging 
tools* 
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INTRODUCTION 

For some applications the native machine is not the system 
of choice in which to develop software, as when the target 
machine is unavailable (because it is still being developed, 
is obsolete, or is inaccessible) or inconvenient (as when 
there is minimal target-system support for debugging). In 
such cases, simulation or emulation may be preferred. Sim¬ 
ulation has the advantage of giving the user intimate access 
to the target machine, usually through a rich debugging 
package. Typically this richness is achieved at a high de¬ 
velopment cost for the simulator and at a target-system 
performance degradation of fpur or more orders of magni¬ 
tude. Emulation can offer processing speeds comparable to 
the target system (even faster, for slow target machines), 
but typically does not support a rich debugging environment. 
In designing the PRIM system, we have attempted to retain 
the best features of both the simulation and emulation ap¬ 
proaches while at the same time minimizing their disadvan¬ 
tages. PRIM provides a sharable, uniform framework for 
running emulations of target machines; within that frame¬ 
work is a rich user interface that supports interactive target- 
system and emulator debugging. When the user is not en¬ 
gaged in debugging, the target system runs at emulator 
speeds, but a sophisticated debugging package is available 
immediately when needed. PRIM was developed within a 
modem timesharing system so as to provide convenient 
access, a file system, resource management, and a large set 
of utilities without the cost of developing yet another oper¬ 
ating system. 

THE PRIM SYSTEM ARCHITECTURE 

The emulation of a target machine under PRIM involves 
three different system levels: a timesharing system, which 
runs on a DEC PDP-10; the PRIM framework, which runs 
at user level on the PDP-10; and target-machine emulation 


* This research was supported by the Defense Advanced Research Projects 
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tools controlled by that framework, which mn on a sharable 
MLP-900 microprogrammable processor. (The MLP-900— 
Multi Lingual Processor—is the prototype of a micropro¬ 
grammable processor designed by Standard Computer Cor¬ 
poration 1 to follow their IC-7000.) This architecture is shown 
schematically in Figure 1. A more complete description of 
the PRIM architecture was presented by the authors at the 
Tenth Annual Workshop on Microprogramming. 2 The 
timesharing system hardware and software provide shared 
access to the MLP-900. The PRIM framework supports in¬ 
teractive users at terminals and provides access to the file 
system for the emulator. And the emulator maintains a com¬ 
plete target-system environment. The PRIM framework can 
be used for both emulator development and target program 
debugging. 

The PDP-10 is a large, general purpose computer to which 
new devices can be connected fairly easily, since the I/O 
bus is extensible and the multiported memory is external to 
the processor. At USC Information Sciences Institute (ISI), 
it runs the TENEX timesharing operating system developed 
at Bolt, Beranek, and Newman. 3 TENEX is strongly ori¬ 
ented toward the support of interactive computing, serving 
both local users and remote users connected via the AR¬ 
PANET. The operating system does not interact directly 
with the user; rather, it allocates resources, manages the file 
system, and supports the execution of TENEX processes, 
each process running in its own paged virtual memory and 
interacting as appropriate with its own user via a terminal 
of some kind. To support PRIM, TENEX was extended 
with software to provide access to the MLP-900 by TENEX 
processes; the MLP-900 was extended with hardware and 
software to guarantee the integrity of TENEX even against 
errant microcode. 

The MLP-900 is a large, fast, vertical-word, micropro¬ 
grammable processor with a writable control memory. The 
processor consists of an operating engine and a control en¬ 
gine. The operating engine is a 36-bit arithmetic/shift unit 
with 32 general registers, 16 mask registers, and a IK inter¬ 
nal memory. The control engine is a control unit with inter¬ 
rupt and branch logic, a subroutine-call stack, 128 pro¬ 
grammable flip/flops, and 4K of writable control memory. 
Cycle time is 250 nanoseconds, during which either one or 
both engines can execute a 32-bit instruction. The MLP-900 
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Figure 1—Architecture of PRIM 


is interfaced as a peripheral device on the PDP-10 I/O bus 
with direct access to the PDP-10 memory via one of the four 
existing memory ports; it has no peripheral devices of its 
own. The I/O bus interface allows the exchange of control 
information between the MLP-900 and the PDP-10; via this 
interface, either processor can interrupt the other. Hardware 
modifications were required only in the MLP-900; they con¬ 
sisted of the interfaces to the PDP-10 and a supervisor/user 
state to protect the I/O bus interface, the MLP pager (an 
address translator in the memory interface that mimics the 
TENEX pager), most of the MLP-900 interrupt system, and 
the MLP-900 control memory itself against user microcode. 

The software at the system level consists of a small op¬ 
erating system resident in the MLP-900, known as the mi¬ 
crovisor, and a new TENEX device driver to shake hands 
with the microvisor and govern access to the MLP-900 by 
TENEX processes. The MLP device driver is the sole 
TENEX operating system module added. It allows a 
TENEX process to create, run, and control a subordinate 
MLP process in much the same way it can a subordinate 
TENEX process. It schedules use of the MLP-900 among 
contending users and supervises the microvisor. Most of the 
microvisor is devoted to swapping emulator contexts (con¬ 
trol memory and MLP registers) as the driver passes control 
of the MLP-900 from one user to another. The process of 
swapping an emulator’s context into and back out of the 
MLP-900 requires about ten milliseconds of microvisor ex¬ 
ecution and introduces no perceptible delays beyond those 
ordinarily encountered in a timesharing system. The rest of 
the micro visor responds to emulator requests for service, 
manages the MLP pager, and performs other tasks required 
by the driver in TENEX. The microvisor runs in the privi¬ 
leged supervisor state that allows access to all resources; 


emulator microcode runs in the user state that protects all 
the critical resources from modification. PDP-10 memory is 
not directly addressed by microcode; instead, memory ref¬ 
erences are to addresses in a virtual memory identical to 
that of a TENEX process. These virtual addresses are trans¬ 
lated to real addresses by the MLP pager, which is con¬ 
trolled (via the microvisor) by the driver in TENEX. A 
reference to a page not in memory results in a page-fault 
interrupt into the microvisor, which passes the fault on to 
the driver and retries the memory operation after the page 
is fetched by TENEX. 

The net effect of the design is a sharable emulation facility 
in which each emulator runs independent of all others in its 
own context, accessing its own virtual target memory under 
control of the PRIM framework that created it. The frame¬ 
work has potential access to all of its emulator’s context 
and target memory and may inspect and/or modify them. 

The PRIM system development started in 1972. The initial 
hardware and software were operational in May 1974 with 
the ability for multiple (local and remote) users to develop 
and debug emulator microcode. The complete system be¬ 
came available in March 1976. All software and hardware 
were developed at ISI. System software took about one 
man-year. The TENEX MLP-900 device driver represents 
about 2000 machine instructions; the MLP-900 micro visor 
occupies about 500 words of control memory.* 


THE PRIM FRAMEWORK 

The PRIM framework consists of TENEX processes that 
define and implement the PRIM user command language, 
create an MLP-900 emulation process and control its exe¬ 
cution at the user’s behest, and provide I/O service for that 
emulation process. The I/O service implements a set of 
primitives that allow the emulator to transfer data to or from 
the TENEX file system. The emulator invokes these pri¬ 
mitives to perform target I/O operations on installed devices 
after the user has associated them with TENEX files. 

An emulator is required to cooperate in the debugging 
process, although the demands are minimal. Essentially, an 
emulator is expected to stop cleanly when interrupted by 
the framework or on the occurrence of a small number of 
predefined events that it monitors and to report its reason(s) 


* It is worth noting that the operating-system level of PRIM was designed, 
developed, and tested using a TENEX system that, except for maintenance, 
was in production use continuously. The hardware interfaces were checked 
out using a dedicated 16K-word memory module and a specially built I/O bus 
simulator that connected the MLP-900 to the PDP-10 via a high-speed EIA 
terminal line. A private MLP-900 driver controlled this “smart terminal.” 
After the microvisor was developed and the swapping and paging algorithms 
validated, the memory module was shared among concurrent MLP-900 users. 
After validation of the hardware and software interfaces between the MLP- 
900 and the PDP-10, the I/O bus simulator was removed and the MLP-900 
was connected to the PDP-10 I/O bus. An MLP-900 device driver was then 
added to TENEX and the MLP-900 connected to the full TENEX memory; 
since most of the driver logic had already been checked out, only a short 
extension of maintenance periods was required for stand-alone checkout of 
the new device driver. The change-over to this driver had no effect on the 
microvisor. 
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for stopping. When the emulator halts or is interrupted by 
user intervention, control returns to the user at command 
level via the framework. Specific commands are available 
to start (or resume) execution. 

The emulator writer must supply the PRIM framework 
with tables that define the target system architecture and 
symbols and drive a target system assembler and disassem¬ 
bler. Except for machine and user symbols and target as- ■ 
sembly language, the same command interactions apply to 
every use of an emulator and framework. The framework 
contains two separate command-processors, known as the 
exec and the debugger. The exec has a large number of 
commands of fixed form that are used infrequently. The 
debugger has a smaller number of commands, but they are 
of variable form and are used much more frequently. Al¬ 
though both command processors offer automatic command 
completion and help facilities, each uses a language tailored 
to its functions. Typically, a user interacts with the exec 
only briefly when starting and ending a session; during the 
session he interacts primarily with a target program or the 
debugger. The exec commands are based on easily remem¬ 
bered keywords that often require several characters to es¬ 
tablish uniqueness; the debugger commands are uniquely 
identified with only a single keystroke. 

The PRIM debugger is a target-system-independent inter¬ 
active program for debugging a PRIM emulator or a target 
program running on such an emulator. Basically, it permits 
a user to examine, search, and modify target system loca¬ 
tions and to set and clear breakpoints. Target system assem¬ 
bly language and symbols are recognized and generated, 
expressions of arbitrary complexity are accepted, and arith¬ 
metic is performed according to the conventions of the target 
machine. The debugger commands are listed in Table I. 

Representation commands allow the user to designate the 
form and notation of the debugger’s representation of data 
(FORMAT, MODE, and the three SYMBOL commands). 
Target system data may be represented as text or instruc¬ 
tions or as formatted or unformatted numeric values; num¬ 
bers may be represented as symbols or as signed or unsigned 
integers. The user may define multi-field data formats, select 
user symbol tables loaded earlier via an exec command, and 
add new symbols to a symbol table or delete existing ones. 

Control commands allow the user to execute debugger 
commands conditionally or indirectly (IF and USE), to con¬ 
tinue execution of a target system, either at the system’s 


TABLE I.—PRIM Debugger Commands 


Representation 

Control 

Display 

Miscellaneous 

FORMAT 

MLP BREAK* 

MLP TYPE* 

MLP CHANGE* 

KILL SYMBOL 

MLP STEP* 

EQUALS 

CLEAR 

MODE 

BREAK 

EVALUATE 

COMMENT 

NEW SYMBOL 

DEBREAK 

JUMP HISTORY 

SET 

OPEN SYMBOL 

GO 

LOCATE 



IF 

NEXT 



PROGRAM EDIT 

PRIOR 



RETURN 

SAME 



SINGLE STEP 

TYPE 



USE 

* Commands for emulator developers only. 


program counter or at a supplied address (GO), to single- 
step a target program (SINGLE STEP), to set and clear 
breakpoints in the target system and enter or edit debugger 
command “programs” to be executed at break time 
(BREAK, DEBREAK, and PROGRAM EDIT), and to re¬ 
turn control to the PRIM exec (RETURN); privileged com¬ 
mands permit an emulator writer to single-step or set or 
clear a break in the MLP-900 (MLP STEP and MLP 
BREAK). Conditional and indirect execution are particu¬ 
larly important when a command is to be executed at a later 
time in an indeterminate context, such as in a command file 
or a break-time program. Breakpoints can be set on refer¬ 
ences (read, write, or execute) to target machine locations 
or on events that have been predefined by the emulator 
writer. The most powerful features of breakpointing is the 
ability for a user to associate with each breakpoint a se¬ 
quence of debugger commands (the break-time program) to 
be executed when the break condition is detected and con¬ 
trol is passed to the debugger. Coupled with conditional and 
indirect execution of debugger commands and the ability to 
continue target system execution, break-time programs offer 
the user a very powerful facility to trace execution, monitor 
events, and perform other complex exploratory functions 
automatically and repeatedly. 

Display commands allow the user to examine the contents 
of designated target locations (TYPE, PRIOR, NEXT, 
SAME, and EQUALS), to search extended areas within the 
target system according to content (LOCATE), to evaluate 
arbitrary expressions (EVALUATE), and to review a record 
of recent target-system control transfers (JUMP HISTORY); 
a privileged command permits an emulator writer to display 
MLP-900 control memory symbolically (MLP TYPE). Any 
displayed location may be modified. 

The miscellaneous commands offer convenience features 
that do not fit in the other categories. They allow the user 
to change the contents of target system locations without 
having to display them first (SET and CLEAR) and to an¬ 
notate transcripts or break-time output (COMMENT); a 
privileged command permits an emulator writer to compile 
code changes directly into MLP-900 control memory (MLP 
CHANGE). 

The PRIM exec comprises a diverse collection of com¬ 
mands that are not directly involved in the intimate control 
of debugging. The exec commands are listed in Table II. 
One major group of commands is concerned with the estab¬ 
lishment of a target machine configuration (INSTALL, SET, 
and UNINSTALL) and with the mounting of TENEX files 
on the configured I/O devices (CANCEL, MOUNT, REAS¬ 
SIGN, REWIND, and UNMOUNT). Another group of com¬ 
mands involves the loading of various classes of state infor¬ 
mation from files (LOAD, TABLES, RESTORE, and 
SYMBOLS) and with the saving of all or part of the target 
context on a file for subsequent restoration (SAVE). There 
are commands that transfer control from the exec to some 
other state (QUIT, DEBUG, and GO), and commands that 
alter the state of the PRIM framework (NO, CHANGE, 
COMMANDS, and ENABLE). PRIM will, on request, 
maintain a transcript of all or a part of a PRIM session 
(TRANSCRIPT and CLOSE); such transcripts have been 
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TABLE II.—PRIM Exec Commands 


Configuration 

Save/Restore 

Miscellaneous 

CANCEL 

LOAD* 

NO* 

FILESTATUS 

TABLES* 

CHANGE 

INSTALL 

RESTORE 

CLOSE 

MOUNT 

SAVE 

COMMANDS 

PERIPHERALS 

SYMBOLS 

DEBUGGER 

REASSIGN 


ENABLE 

REWIND 


GO 

SET 


NEWS 

SHOW 


QUIT 

UNINSTALL 


TIME 

UNMOUNT 


TRANSCRIPT 


* Commands for emulator developers only. 


found invaluable by users with CRT terminals—and there¬ 
fore no written record of their work. The remaining com¬ 
mands (FILESTATUS, PERIPHERALS, SHOW, NEWS, 
and TIME) are information-displaying commands. 

The PRIM framework was written in BLISS. 4,5 It occupies 
about 35K words of PDP-10 memory, including fixed work 
space, and took somewhat over three man-years to produce. 

EMULATOR MODEL 

A prototypical PRIM emulator has been developed based 
on the constraints of the MLP-900 environment, the objec¬ 
tives of the PRIM system, the requirements of the PRIM 
framework, and the specific interface conventions that 
framework defines. 6 The environment consists of execute- 
only microcode residing in control memory, the MLP-900 
registers, and a 256K 36-bit (virtual) main memory; the reg¬ 
isters and virtual memory together comprise the context into 
which are mapped the registers and memory of the target 
machine plus various other regions devoted to required 
PRIM functions. The mapping is arranged at the conve¬ 
nience of the emulator writer, with accompanying tables de¬ 
scribing this mapping to the PRIM framework. The emulator 
can modify its context in the course of emulation, stop 
(thereby returning control to the framework), and request 
I/O services from the framework. 

The most fundamental PRIM requirement is for a bit- 
compatible emulation of the target machine with accurate 
timing. Beyond that, an emulator must have a control struc¬ 
ture that allows it to stop after any cycle and subsequently 
resume emulation in a manner totally transparent to the 
target machine. While a single target instruction constitutes 
the typical cycle, other events, such as interrupts or I/O 
data transfers, must also be treated as emulator cycles. 
Changes to the context made by the user during an emulator 
stop must appropriately affect the target machine on re¬ 
sumption of emulation. In particular, this requires that the 
emulator have no hidden copies of target state information 
when it stops. 


Target timing in PRIM is virtual. The emulator is required 
to increment an internal, high-resolution (typically, 50 na¬ 
nosecond) clock to reflect the passage of target-machine 
time; there is no fixed relationship among target time, MLP- 
900 time, PRIM framework time, and real time. Emulated 
cycles that consume target time (e.g., instruction execution) 
advance the clock; emulated cycles that nominally occur in 
parallel with the former (e.g., I/O controller activity) are 
scheduled for service relative to that internal clock. The 
result is a small, event-driven, discrete simulation system 
with target instruction execution being treated as a back¬ 
ground task. Figure 2 is a top-level outline of a typical PRIM 
emulation tool. 

Target peripheral devices are included in an emulation. 
Device control logic, timing, and interaction with the target 
machine are contained entirely within the emulator. An I/O 
transfer is performed by the I/O server in the PRIM frame¬ 
work in response to a call by the emulator, which must then 
monitor the transfer for completion. The transfer is carried 
on in parallel with emulator execution, and the emulator 
may have several I/O operations outstanding simultane¬ 
ously. The freedom given the user to define target system 
configurations and the protocols for use of the I/O server 
primitives, together constitute a generalized framework for 
device emulation. 

User commands to inspect and modify the target machine 
environment are implemented by the PRIM framework via 
tables associated with each emulator, requiring no direct 
intervention by the emulator itself. But user commands in¬ 
volving the breakpoint facility do require the cooperation of 
the emulator. Each breakpoint a user sets is marked by a 
flag in a portion of the shared context outside the address 
space of the target machine; these flags are called “meta¬ 
bits.” The emulator must monitor the target machine con- 


initialixe emulator ; 

FOREVER DO 
BEGIN 

IF reason to stop 

THEN BEGIN 
stop; 

respond to switches and buttons ; 
END ; 

IF time to serve next scheduled event 
THEN BEGIN 

remove event from list ; 
execute event routine ; 

END 

ELSE 

IF interrupt pending 
THEN execute interrupt cycle 
ELSE execute instruction cycle ; 
END; 


Figure 2—PRIM emulator outline 
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tinually for the occurrence of any breakpoints flagged by 
these meta bits and, when one occurs, log it and prepare to 
stop at the end of the current emulation cycle. Interrupting 
execution at the end of the cycle is consistent with the 
requirement for stopping cleanly and also avoids a problem 
that would arise if execution were interrupted in mid-cycle 
on the occurence of the break condition—that of responding 
to possible changes made in the context during the break 
while not reporting the same breakpoint repeatedly. 

PRIM defines two classes of breakpoints, known as ref¬ 
erence breaks and event breaks. A reference break occurs 
on a read (data fetch), write, or execute reference to spec¬ 
ified target locations. The meta bits for reference breaks are 
the four high-order bits of each 36-bit context word. As 
every target location can have its own meta bits, there is no 
practical limit on the number of simultaneous breakpoints 
the user may set. An event break occurs whenever a spec¬ 
ified target-machine event arises. Each emulator supports 
its own set of event breaks, but at a minimum includes 
instruction execution (target machine single-step), jumps or 
branches, memory stores, interrupts, and I/O transfers. One 
meta bit is used for each type of event break an emulator 
monitors. The event breaks are particularly useful for close 
monitoring of target machine behavior. 

To date, there are three supported PRIM emulations: the 
AN/UYK-20 (a modern 16-bit computer), 7 the Univac U1050 
(an ancient byte-addressed decimal machine with a 30-bit 
instruction word), 8 and the Intel 8080 chip. Both the AN/ 
UYK-20 and the U1050 emulations include a comprehensive 
set of peripherals. These two emulators each require about 
3K words of control memory; the 8080 emulator, which 
supports only a simple terminal-like device, requires about 
IK words of control memory. For the AN/UYK-20, which 
is a well-documented system of moderate complexity, the 
emulation of the CPU and the peripherals each required 
about six man-months. Although the U1050 is of comparable 
complexity, development of the emulator took somewhat 
longer because of the unavailability of accurate and up-to- 
date documentation. The 8080 emulation took only one man- 
month. All of these emulators require on the order of five 
microseconds to emulate one target memory cycle. For the 
AN/UYK-20, the ratio of emulated-to-real time is about 
10/1; for the U1050 and the 8080, the ratio is about 1/1. For 
all emulations, as the I/O traffic increases, the performance 
ratio increases. 


DISCUSSION 

Since it became operational, PRIM has been used exten¬ 
sively by the System Design Laboratory (SDL) at the Naval 
Ocean Systems Center in San Diego, by the Logistics group 
at Gunter Air Force Station, and by a “smart terminal” 
project at ISI. SDL is using PRIM to support the design and 
development of naval systems employing the AN/UYK-20 


minicomputer. The Logistics group at Gunter has used 
PRIM to examine the consequences of replacing existing 
peripherals with higher-performance ones on a Univac 
U1050 system. In the process, they discovered an unex¬ 
pected design limitation in a communications interrupt rou¬ 
tine and also uncovered several system bugs for which 
patches were developed on the spot. An Intel 8080 emulation 
was used at ISI to debug terminal software that would oth¬ 
erwise have been impossible to observe. The MLP-900 has 
been used an average 20 hours per month (representing an 
average of about 40 hours of connect time) by as many as 
six concurrent users. During this time an average of ten 
hours of TENEX time per month was used by the PRIM 
framework. The maximum level of use to date has been 90 
hours of MLP-900 time in one month. Ordinarily, a PRIM 
user is unaware that he is sharing the MLP-900 with other 
users. 

By cleanly and sharply separating the debugging and tar¬ 
get-machine emulation tasks, PRIM has been able to avoid 
most of the disadvantages of simulation and emulation while 
at the same time combining their advantages. In achieving 
this sharp separation of function, PRIM established a uni¬ 
form and systematic structure for the development of emu¬ 
lators. This structure not only minimizes the involvement of 
the emulator in the debugging process, but also greatly sim¬ 
plifies the task of emulator development as it utilizes a 
standard package of I/O service routines and provides a 
convenient control structure suitable for a large family of 
target-machine emulations. As most of PRIM consists of 
sharable system-level and user-level code that is common to 
this potentially large family of target system emulations, a 
more extensive development effort (with its consequently 
more sophisticated design) could be justified than would 
have been appropriate for a single-machine emulation or 
simulation. 
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A microprogrammed AN/UYK-20(V) emulation* 


by DONALD A. DEEL and WALTER A. BURKHARD 

University of California 
La Jolla, California 

INTRODUCTION 

In this paper an emulation of the Sperry Univac AN/UYK- 
20 computer on the Nanodata QM-1 2 is described; the paper 
is divided into four sections. Short descriptions of both the 
target machine and the host machine are presented in the 
first section, each emphasizing those characteristics which 
most affected the emulation. The second section consists of 
a description of the basic structure of the emulation. This 
includes brief explanations of the flow of control through 
the various parts of the emulation, the principal routines, 
and the allocations of storage within the QM-1. Also, inac¬ 
curacies in the emulation are noted, with an indication of 
their effect and how they might be remedied. The third 
section gives a sampling of emulation performance charac¬ 
teristics and in the final section future plans are presented. 

The AN/UYK-20 target machine 

The AN/UYK-20 is a militarized 16 bit minicomputer. It 
possesses a large instruction set and good I/O provisions. 

The following paragraphs form a brief description of the 
machine. 

Memory 

Main memory consists of up to 64K words of 16 bits each. 
Several memory locations are dedicated to specific pur¬ 
poses: locations 110-137 (octal) are for processing interrupts; 
locations 140-141 (octal) are the I/O command cells, which 
are used to start up I/O transfers; and locations 200-217 are 
for external interrupt word storage. 

Registers 

There are two banks of general purpose registers, with 
sixteen 16 bit registers in each bank. Only one bank can be 
in use at one time; which bank is currently in use is deter¬ 
mined by a bit in the Status Register #1. This register also 
indicates which interrupt classes are locked out, what the 
current condition code is, and what the carry and overflow 

* This work supported in part by NOSC Contract No. 66001-77-C-018. 


bits are. The Status Register #2 controls the type of ad¬ 
dressing that will occur for certain kinds of memory refer¬ 
ences. It also holds I/O instruction fault and memory resume 
interrupt data. 

Other registers include a Real Time Clock, a Monitor 
Clock, and 64 Page Registers. The Real Time Clock is a 32 
bit register which counts up, and the Monitor Clock is a 16 
bit register which counts down to zero. The Page Registers 
are used to affect all memory accesses. The upper 6 bits of 
each 16 bit address coming from the CPU is used to access 
one of the 64 Page Registers; the six-bit contents of the 
selected page register is then substituted for the original 
upper 6 bits of the memory address. 

Instruction formats 

The AN/UYK-20 has both single and double word instruc¬ 
tions. Figure 1 presents the various formats. 

I/O provisions 

There are sixteen I/O channels; each with dedicated con¬ 
trol memories. Figure 2 displays the data formats within 
each I/O channel control memory. 

A Control Memory holds all the information required by 
the I/O channel to perform data transfers. I/O instructions 
that control the input and output activity on all peripheral 
channels are executed under an active chain that is associ¬ 
ated with each input and each output channel. I/O activity 
is initiated in the channels by placing the “initiate chain” 
instruction in the command cells in main memory (locations 
140-141) and executing the “I/O command” CPU instruc¬ 
tion. 

The I/O Channels are capable of performing 8 bit or 16 
bit parallel data transfers. 32 bit transfers can be affected by 
using two channels wired together. Serial I/O channels are 
available which meet the characteristics specified in EIA 
standard RS-232C, MIL-STD-188C, or NTDS serial speci¬ 
fication. 

Interrupts 

There are three classes of AN/UYK-20 interrupts. From 
highest to lowest priority, these are: hardware error inter¬ 
rupts; software interrupts; and I/O interrupts. Within each 
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TYPE 15 14 13 12 11 10 9 8 7 < 5 4 3 2 1 0 



DEFINITION OF FIELDS 

OPCOOE: CODE SPECIFYING THE OPERATION 

f: FORMAT DESIGNATOR 


«-00 FORMAT RR REGISTER REGISTER OR RL 1 FORMAT 

- 01 FORMAT Rl REGISTER IMMEDIATE MEMORY OR HL 2 FORMAT 

- 10 FORMAT REREGISTER LITERAL CONSTANT OR RL 3 FORMAT 

-11 FORMAT RX REGISTER INDEXED ADDRESS OR CONSTANT OR RL-4 FORMAT 

a: GENERAL REGISTER OR SU8FUNCTION DESIGNATOR 

mi GENERAL REGISTER OR SUBFUNCTION DESIGNATOR 

in: 4 BIT UNSIGNED LITERAL IN RL FORMAT 

4: DEVIATION (VALUE) 

a: SIGN DESIGNATOR FOR d 

0-POSITIVE 
1 • NEGATIVE 

y: AOORESS OR ARITHMETIC CONSTANT 

Figure 1—AN/UYK-20 instruction formats 

class, there is a priority structure to handle simultaneous 
interrupts. 

Each interrupt class has associated with it several dedi¬ 
cated locations in main memory. These locations are similar 
to the OLD PSW/NEW PSW memory allocations in the 
IBM 360. Interrupts are responded to by first storing the 
program counter, status registers, and Real-Time Clock in 
several of the dedicated locations, and then re-loading these 
registers from the remaining dedicated locations. 


The QM-1 host machine 

The Nanodata QM-1 is a machine possessing two levels 
of microprogrammed control. This is accomplished using 
three different memories: Nanostore, Control Store, and 
Main Store. Target Machine (AN/UYK-20) instructions in 
Main Store are executed by (and defined by) microprograms 
in Control Store, under vertical control. Microinstructions 
in Control Store are in turn executed (and defined by) Nan¬ 
oprograms in Nanostore, under horizontal control. This 
Control Hierarchy is illustrated in Figure 3. 

With appropriate software, including both a micro lan¬ 
guage definition and systems support functions for the QM- 
1, the micro-level user can utilize the machine without being 
concerned with the nano-level. The micro language defini¬ 
tion consists of a complete set of nano programs collectively 
called MULTI, 3 which is well-adapted to writing emulations. 
The systems support is given by two programs written in 
the MULTI micro language called TASK 4 and PROD. 5 
TASK supports multiple co-resident Control Store tasks in 
a hierarchical manner. TASK also manages all QM-1 I/O 
functions for the micro-programmer. PROD (Programmable 
Run-time Operator’s Display) allows the QM-1 operator to 
use the console to control the emulated target machine. 
PROD also provides trace/debug capabilities for the emu¬ 
lated hardware system and trace-debug capabilities for the 
microprogrammer. Since MULTI, TASK, and PROD were 
used for the AN/UYK-20 emulation, the Nanoprogramming 
level of the QM-1 will not be discussed in this paper. 

The following paragraphs are a brief overview of the QM- 
1 resources available to the microprogrammer using 
MULTI. 



Control store 

Control Store is a fully readable/writable general-purpose 
18 bit wide store implemented in 75 ns semiconductor mem¬ 
ory. A maximum control store configuration consists of 16K 
words. 

Main store 

Main Store is a general-purpose 18 bit core storage con¬ 
sisting of up to 256K words; full cycle time is 800 ns. 



fetched and 
nanoprogram 


Nanoprograms are directly 
executed by hardware 


Nanoprogram executing 
"XYZ" is shaded 


Figure 3—QM control hierarchy 
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Local store 

Local Store is a bank of 32 18 bit registers. Some of these 
registers have special uses; for instance, the micro program 
counter and the micro instruction register are kept in Local 
Store. 

Arithmetic-logic unit 

The Arithmetic-Logic Unit can be controlled to perform 
all of the 16 logical (Boolean) operations, as well as common 
arithmetic operations, upon two 18 bit operands. A 16 bit 
mode permits the two input operands to be sign-extended 
from 16 to 18 bits so that the operation of the ALU need 
not be changed when dealing with 16 bit data values. Also, 
a decimal control facilitates decimal arithmetic. 

Shifter 

The 18 bit combinational logic shifter is capable of per¬ 
forming both left and right logical and circular shifts, as well 
as right arithmetic shifts. Also, double precision (36 bit) left 
and right logical, arithmetic, and circular shifts can be spec¬ 
ified. 

THE EMULATION OF THE AN/UYK-20 ON THE 

QM-1 

The emulation of a machine on the QM-1 is very straight¬ 
forward. It consists of an assembly language program run¬ 
ning at the Control Store level of the QM-1. This micro¬ 
program explicitly performs all the operations and data ma¬ 
nipulations expected of the emulated computer. Thus, object 
code (for the emulated machine) residing in the Main Store 
of the QM-1 can be executed in the same manner as it would 
be in the emulated (target) machine. This micro-program 
should be considered to be an “action translates ” it con¬ 
verts each of the emulated machine’s action requests (the 
object code instructions) into an equivalent set of QM-1 
actions. 

In the case of the AN/UYK-20, the micro-program that 
runs in Control Store has several I/O channels to emulate as 
well as the Central Processor itself. The micro-program per¬ 
forms the following four operations repeatedly: execute one 
AN/UYK-20 micro-instruction; service each emulated I/O 
channel as necessary; update the emulated real-time clocks; 



Figure 4 —AN/UYK-20 emulation cycle 


emulate any interrupts that might have been caused by the 
previous three items in this list. Figure 4 illustrates this 
structure. 

The following paragraphs are a brief description of the 
control flow through the various sections of the main emu¬ 
lator loop. 

The CPU emulation 

The part of the emulation that executes one AN/UYK-20 
micro-instruction begins by reading the instruction out of 
Main Store. The number in the emulated program counter 
is used as the address. Once the instruction has been ob¬ 
tained, the six-bit opcode field is inspected to determine 
whether it is a Central Processor command, or an I/O in¬ 
struction. If it is an I/O instruction, then an error flag is 
immediately set that tells the interrupts emulation to gen¬ 
erate the equivalent of the AN/UYK-20 Central Processor 
instruction fault interrupt; the flow of control then passes to 
the interrupts emulation. 

For Central Processor commands, it is normal to decode 
the opcode of the instruction at this point and then execute 
some micro-routine that is equivalent to the instruction. In 
the AN/UYK-20, however, it is possible to pre-fetch at least 
one of the instruction’s operands before decoding the op¬ 
code. This is done by a micro-routine which first determines 
the type of the instruction from the two-bit f field. Then, 
based upon this information, an operand register kept in the 
Local Store of the QM-1 is loaded with the appropriate data 
item which always comes either from the currently active 
AN/UYK-20 general-purpose register bank or from the AN/ 
UYK-20 memory. All of the various memory reference 
modes are handled in this pre-fetch routine; these modes 
include direct, indexed, indirect, and cascaded indirect 
memory references. Besides the operand, the routine also 
returns the four-bit A and M fields of the instruction in QM- 
1 Local Store registers; these parameters are used very 
frequently in processing an instruction after it has been 
decoded. Also, the address of the operand is put into Local 
Store if the operand is from memory. This address is used 
by the “store” instructions. 

After the operand is pre-fetched, the instruction is de¬ 
coded. This is accomplished by using the six-bit opcode and 
the two-bit modifier field (the f field) together as an index 
into a table of 256 Control Store locations. Each entry in 
this table is the Control Store address of a micro-routine 
which will execute an AN/UYK-20 micro-instruction. Op¬ 
code-modifier combinations which are undefined in the AN/ 
UYK-20 all reference the address of a routine that imme¬ 
diately generates an emulated Central Processor instruction 
fault interrupt. 

Immediately before entering a micro-routine, the QM-1 is 
switched to the 16 bit mode; consequently, the QM-1 ALU 
conditions can be used directly to set the AN/UYK-20 con¬ 
ditions code. All other portions of the emulation utilize the 
full 18 bit width of the QM-1 system. 

The micro-routines which actually “execute” the instruc¬ 
tion generally only consist of a few micro-instructions that 
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manipulate the QM-l’s resources to implement the instruc¬ 
tion. Routines that do not affect the AN/UYK-20 conditions 
code switch the QM-1 back to 18 bit mode and then exit to 
the next portion of the emulation. Routines that set the 
conditions code exit to one of two conditions-setting rou¬ 
tines (single and double precision) before setting 18 bit mode 
and going to the next emulation section. 

The I/O emulation 

Two of the possible sixteen I/O channels of an AN/UYK- 
20 are presently emulated. This is because two channels are 
sufficient to handle the peripheral devices available to the 
emulation: a card reader, a lineprinter, and a CRT console. 
Additional channels can easily be added for more devices. 

The emulation maintains a special flag which indicates 
whether any I/O activity is taking place. If this flag is not 
on, then the I/O emulation is exited immediately. This mech¬ 
anism allows the CPU emulation to run as rapidly as is 
possible. Otherwise, the channels are emulated serially from 
highest to lowest priority. 

Each emulated channel has a Control Store word of flag 
bits associated with it that reflect the current status of the 
channel. AN/UYK-20 channels have both input and output 
parts that are capable of executing I/O instructions or per¬ 
forming data transfers at the same time. When a channel is 
being emulated, the micro-program checks both parts to see 
if they are executing some I/O chaining program that is 
either active or waiting. Active chaining programs, or 
chains, are serviced by executing one I/O chaining command 
from memory. Waiting chains are waiting for some I/O trans¬ 
fer to complete, and are serviced by setting the chain “Ac¬ 
tive” when the transfer is finished. When an input or output 
portion of a channel is neither active nor waiting, it is dor¬ 
mant and requires no servicing. 

AN/UYK-20 Channels have Control Memories; in the em¬ 
ulation, these are kept in Control Store. The Control Mem¬ 
ory of a channel holds information needed for both the input 
and output portions such as: a buffer address, a word count, 
and a chain address pointer, which holds the address of the 
next I/O command to be executed if that portion of the 
channel is active. The entire I/O interpreter is structured as 
a subroutine so that each channel with an active chaining 
program could easily be serviced. This routine begins by 
fetching an I/O command from memory. The chain address 
pointer in the channel’s control memory is used as the 
address. After the fetch, the chain address pointer is incre¬ 
mented to point at the next instruction. The opcode is then 
inspected; if it is not a valid I/O Command, control imme¬ 
diately passes to a micro-routine that generates an emulated 
I/O Instruction fault interrupt. Otherwise, the instruction’s 
operand is pre-fetched. The major difference between the 
I/O pre-fetch routine and the Central Processor pre-fetch rou¬ 
tine is that there are no indirect references in I/O Com¬ 
mands. After the pre-fetch is done, the instruction is de¬ 
coded via a table of micro-routine addresses. As in the 
Central Processor emulation, undefined opcodes result in 
entries in the table that all point to one routine that generates 


an emulated I/O Instruction fault interrupt. Each micro-rou¬ 
tine that actually executes the I/O command ends by return¬ 
ing to the routine that called the I/O command interpreter 
subroutine. 

Emulation of the real-time clocks 

The emulated Real-Time Clocks do not run in true real 
time; they are run at the same speed as the rest of the 
emulation. The actual execution times of Central processor 
instructions and I/O commands are well-known; these times 
make up the entries in a Control Store table. Just after an 
construction has been executed, this table is referenced, and 
the entry corresponding to the last instruction is added to 
the emulated Real-Time Clock whenever it is enabled. Sim¬ 
ilarly, the emulated Monitor Clock is decremented by the 
entry if the Monitor Clock is enabled. When the 16 bit 
Monitor Clock goes to zero, the emulated Monitor Clock 
interrupt is generated. When the lower 16 bits of the 32 bit 
Real-Time Clock overflows, the emulated Real Time Clock 
interrupt is generated. Running the Real-Time Clocks in this 
“relative” time keeps events occurring in the emulation in 
the same relative order that they would have had in the AN/ 
UYK-20 itself. 

Emulation of the interrupts 

All interrupts are emulated in a single routine so that the 
proper priority scheme can be maintained. The Central Pro¬ 
cessor, I/O, and Real-Time Clock emulations generate in¬ 
terrupts by setting the appropriate bits in a QM-1 Local 
Store register. 

The interrupts emulation proceeds by checking each of 
the enabled AN/UYK-20 interrupt classes from lowest to 
highest priority. Within each enabled class, each of the ap¬ 
plicable emulation interrupt flags are checked, from highest 
to lowest priority; when an interrupt flag is found on, it is 
reset and the interrupt is immediately emulated. No further 
checking of interrupt flags is done for that class. This proc¬ 
ess of checking for interrupt flags is then repeated for the 
next higher priority enabled class of interrupt. 

Interrupts are emulated by implementing the Old PSW/ 
New PSW interrupt structure of the AN/UYK-20. When an 
interrupt must be honored, the emulation stores the AN/ 
UYK-20 program counter, status register #1, status register 
#2, and the Real-Time Clock in Main Store locations dedi¬ 
cated for that purpose. These registers are then loaded from 
an adjacent set of Main Store locations that contain what is 
commonly called the “New PSW”; this causes the desired 
interrupt response, since the Program Counter has been 
changed. The emulation also calculates an interrupt entrance 
address index for the class and type of interrupt being hon¬ 
ored and adds it to the new program counter. 

Simultaneous interrupts of different AN/UYK-20 classes 
result in Old PSW/New PSW switching in order from lowest 
to highest priority; then the individual interrupts are serviced 
in order from highest to lowest priority by AN/UYK-20 
programs in Main Store. 
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Inaccuracies in the emulation 

The emulation of the AN/UYK-20 has the following in¬ 
accuracies: 

The mapping of the upper 6 bits of AN/UYK-20 memory 
addresses through the 64 Page Registers does not occur. 
The Page Registers themselves exist and can be loaded 
and read up normally. No real difficulty is foreseen in 
correcting this; the nanoprograms have been written but 
not implemented or tested. 

The nondestructive read-out (NDRO) memory is not em¬ 
ulated. There are no plans to correct this, since the AN/ 
UYK-20 Bootstrap programs can reside in Control Store. 
Thus, the NDRO Mode Designator bit in Status Register 
# 1 has no effect whatsoever. 

These are the only known discrepancies in the emulation 
of the basic AN/UYK-20 at this time. The I/O emulation is 
complete where the following peripherals are concerned: the 
card reader, lineprinter, and CRT console. 


PERFORMANCE 

The AN/UYK-20 emulation is not large; it occupies less 
than 2.5K words of 18 bit Control Store. Additionally, 
TASK and PROD occupy approximately 2K words of 18 bit 
Control Store and 4K words of 18 bit main store. 

One of the primary concerns with an emulator is its run¬ 
time efficiency. With no I/O activity in progress, the AN/ 
UYK-20 emulation executes CPU instructions with an av¬ 
erage slowdown factor of approximately 25. This number 
was determined by first finding the total time required by 
the QM-1 to execute an entire emulator cycle for several 
AN/UYK-20 instructions. These times were then compared 
to the AN/UYK-20 execution times for the same instruc¬ 
tions. 

Representative times for typical AN/UYK-20 instructions 
under different instruction formats are presented in Appen¬ 
dix A. It is assumed that no I/O activity is taking place and 
that no interrupts are pending. Each instruction time is bro¬ 
ken down into the times taken to go through each section of 
the emulation; each time represents the worst case time for 
that section. The time units are QM-1 T-periods, which are 
80 nanoseconds each. 

From these examples, it should be clear that there is an 
emulation overhead of at least 269 T-periods for every AN/ 
UYK-20 CPU instruction executed. This is incurred by the 
code sections that fetch instructions, decode instructions, 
check for I/O activity, check for interrupts and check for 
emulator single-step mode. The only sections that vary in 
execution time are those which pre-fetch operands, execute 
instructions, and set conditions (some instructions do not 
set conditions). It is this emulation overhead which gives 
rise to the rather severe slowdown factor of 50.5 in the case 
of the RR format Load instruction. The overhead time of 
269 T-periods constitutes more than half of the execution 
cycle time of 474 T-periods. This is not true in the case of 


RX format instructions and instructions which take longer 
to execute on the AN/UYK-20. This can be seen in the RX 
format Load and the two Load PSW instructions, which 
have slowdown factors of 19.3, 13.3 and 11.6, respectively. 
The last instruction timing breakdown is given as an example 
of an instruction where the emulation overhead is not a 
dominate factor. 

The overhead incurred by checking for I/O Activity, In¬ 
terrupts, and Single-Step Mode can be reduced by an order 
of magnitude. This will result in a 15 percent reduction in 
the time required to execute an AN/UYK-20 instruction. 

FUTURE PLANS 

Currently, the AN/UYK-20 emulation on the QM-1 cor¬ 
rectly executes the entire Univac AN/UYK-20 Processor 1 
Diagnostic test program. 6 This thoroughly tests the CPU 
functions. Also, several I/O tests have been run in which 
the emulation has had all the connected peripherals busy 
while simultaneously executing normal CPU instructions. 

Currently, the emulation can write and read Main Store 
images to and from the QM-1 tape drives; the Processor 1 
Diagnostic is loaded into memory for execution in this man¬ 
ner. Adding true bootstrap ability will allow other large 
programs to be easily loaded. This will entail emulating 
another AN/UYK-20 I/O Channel for the tape drive and 
putting the appropriate AN/UYK-20 bootstrap code in Con¬ 
trol Store. 

One of the first “larger” programs to be run on the em¬ 
ulation will be the AN/UYK-20 Level II Operating System. 
This will allow the QM-1 to be used as a complete, stand¬ 
alone AN/UYK-20 system. It will also require the addition 
of an AN/UYK-20 disk I/O Channel to the emulation. 

The completed emulation can be used for software devel¬ 
opment. It is well suited to this task, since an operator can 
exercise much greater control over the emulation than over 
a hardware AN/UYK-20. 
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APPENDIX A—TYPICAL AN/UYK-20 EMULATION 
TIMINGS 

AN/UYK-20 opcode 01: LOAD (RR format) 

T-periods 

108 Instruction Fetch 
86 Operand Pre-fetch (RR format) 

63 Instruction Decode and Update RTC 
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15 

Instruction Execution 

104 

Set Conditions Code 

28 

Check for I/O Activity 

55 

Check for Interrupts 

15 

Check for Emulator Single-Step Mode 

Total = 474 T 

=periods =37.9/as 

Actual AN/UYK-20 execution time=0.75/AS 


Slowdown factor=50.5 
AN/UYK-20 opcode 01: LOAD (RX format) 


T-periods 


108 

Instruction Fetch 

155 

Operand Pre-fetch (RX format with no 
indexing) 

63 

Instruction Decode and Update RTC 

15 

Instruction Execution 

104 

Set Conditions Code 

28 

Check for I/O Activity 

55 

Check for Interrupts 

15 

Check for Emulator Single-Step Mode 


Total = 543 T-periods =43.4/as 

Actual AN/UYK-20 execution time=2. 25/as 

Slowdown factor= 19.3 

Note: If indexing is done in the Operand Pre-fetch, the total 
time goes up by 92 T-periods; this makes the total 
time=635 T-periods=50.8/AS and the slowdown factor 
becomes 22.6. 

AN/UYK-20 opcode 07: Load OSW (RI format) 


T-periods 

108 

Instruction Fetch 

110 

Operand Pre-fetch (RI format) 

63 

Instruction Decode and Update RTC 

120 

Instruction Execution 

0 

No conditions are set 

28 

Check for I/O Activity 


55 Check for Interrupts 

15 Check for Emulator Single-Step Mode 

Total = 499 T-periods=39.9/AS 
Actual AN/UYK-20 execution time = 3.0/as 
Slowdown factor=13.3 

AN/UYK-20 opcode 07: Load PSW (RX format) 


T-periods 


108 

Instruction Fetch 

155 

Operand Pre-fetch (RX format with no 
indexing) 

63 

Instruction Decode and Update RTC 

120 

Instruction Execution 

0 

No conditions are set 

28 

Check for I/O Activity 

55 

Check for Interrupts 

15 

Check for Emulator Single-Step Mode 


Total = 544 T-periods=43.5/AS 

Actual AN/UYK-20 execution time =3.75/as 

Slowdown factor = 11.6 

Note: If indexing is done in the Operand Pre-fetch, the 
Slowdown factor becomes 13.6. 

AN/UYK-20 opcode 17: Circular Left Double Shift (RK 
format) 

T-periods 

108 Instruction Fetch 

123 Operand Pre-fetch (RK format with no 
indexing) 

63 Instruction Decode and Update RTC 

748 Instruction Execution 

104 Set Conditions Code 

28 Check for I/O Activity 

55 Check for Interrupts 

15 Check for Emulator Single-Step Mode 


Total = 1244 T-periods =99.5/as 
Actual AN/UYK-20 execution time=3.0//,s 
Slowdown factor=33.2 
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INTRODUCTION 

The current importance of computer simulation of dynamic 
systems is well-known. Such simulation is useful in system 
analysis, design, and testing, as well as in the training of 
human operators. Whenever a computer simulation involves 
interaction with actual system hardware (including humans) 
the computation must by definition take place in real-time; 
i.e., the computer must solve the simulation equations at 
the same speed with which the actual dynamic system op¬ 
erates. Real-time simulation of dynamic systems plays an 
important role in many areas of technology, including aer¬ 
ospace, process control, automotive, and other applications. 
In the simulation of flight vehicles such as aircraft, helicop¬ 
ters, and missiles the computational requirements associated 
with real-time simulation can become especially severe due 
to the complexity of the equations to be solved and the high 
frequencies present in the dynamic behavior of the solutions. 

In the early 1950’s, when real-time simulation of flight 
vehicles began to be achieved on a widespread basis, only 
analog computers possessed the necessary dynamic capa¬ 
bility to accomplish the simulation. Over the ensuing two 
decades, as digital computers have increased dramatically 
in speed and computing power, more and more of the real¬ 
time simulation of flight vehicles (and other dynamic sys¬ 
tems) has been taken over by digital computers. For ex¬ 
ample, today even general purpose digital minicomputers 
have the capability of real-time simulation of many aircraft. 

However, there continue to be many flight-vehicle sys¬ 
tems which currently are impossible or uneconomical to 
simulate adequately in real-time with existing general-pur¬ 
pose digital computers, e.g., small high-performance mis¬ 
siles, high performance aircraft which include flight-control 
system dynamics, helicopters, and jet engines. In many of 
these cases the simulation is being performed successfully 
with hybrid (i.e., combined analog/digital) computing sys¬ 
tems. However, because of the potential advantages which 
all-digital simulation can have over hybrid simulation, e.g., 


better reliability, easier programming, lower cost and oper¬ 
ation from a stored program, there is ever increasing pres¬ 
sure in favor of all-digital simulation. The purpose of this 
paper is to describe a multiprocessor digital computer with 
architecture and hardware designed specifically for efficient 
high-speed simulation of continuous dynamic systems. 


REQUIREMENTS FOR REAL-TIME SIMULATION 

There are a number of requirements associated with real¬ 
time simulation of continuous dynamic systems which 
should be understood to appreciate the problems peculiar to 
such simulation. First of all, because of the basic origin of 
the requirement, a real-time simulation will involve inter¬ 
action with the real world external to the computer, usually 
on a more or less continual basis. For example, a computer 
used for simulation in flight training or for man-in-the-loop 
engineering studies will receive continuous inputs from the 
cockpit controls being activated by the pilot and will need 
continuously to furnish outputs to drive the various cockpit 
displays. When portions of a large simulation are imple¬ 
mented by using the actual system hardware with the re¬ 
maining simulation accomplished on the computer, again the 
hardware furnishes continuous inputs to and requires con¬ 
tinuous outputs from the computer. Of course, in the case 
of digital simulation continuous data will be approximated 
by discrete time series, but in any case the data-word rate 
for each input and output variable must be fast enough to 
avoid compromising the fidelity of the simulation. Thus real¬ 
time simulation often involves high input/output data rates. 
In typical aerospace simulations these rates may exceed 
several hundred data words per second for each of 100 
channels or more. 

In cases where the speed of a digital computer used for 
real-time simulation is being pushed in order to achieve 
satisfactory dynamic accuracy, it almost certainly follows 
that a fixed time step for numerical integration must be used. 
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This also simplifies greatly the A to D (analog-to-digital) and 
D to A (digital-to-analog) conversion problem when the com¬ 
puter is interfaced to external analog channels. 

Many simulations of continuous dynamic systems involve 
evaluation during each integration step of numerous func¬ 
tions of many variables. For example, the aerodynamic coef¬ 
ficients used for flight simulation can involve several dozen 
functions of two, three, four, or more variables with a cor¬ 
respondingly large data base required for function storage. 
Such simulations also require numerous coordinate trans¬ 
formations during each integration frame, i.e., the transfor¬ 
mation of vector components from one spacial coordinate 
system to another. 

Another characteristic of many continuous system simu¬ 
lations is the existence of a wide range of dynamic frequen¬ 
cies inherent in the system. For example, a typical air frame 
simulation might exhibit a phugoid longitudinal mode at 0.01 
hertz and at the same time have a short period pitching 
mode of 1 hertz, a structural vibration mode of 10 hertz, and 
a control-surface actuator mode of 25 hertz. The integration 
frame rate must be high enough to handle the highest mode 
frequency with adequate accuracy. This means that the in¬ 
tegration frame rate is extremely high (i.e., step size is 
extremely small) with respect to the mode corresponding to 
the lowest frequency. This can lead to errors caused by 
roundoff if care is not taken. Multiple integration frame rates 
can also be employed, but most users prefer to avoid that 
complexity. 

Finally, typical continuous system simulations often in¬ 
volve nonlinear functions with discontinuities, especially in 
the modeling of control systems. Typical examples include 
linear functions with limiting, relay (on-off) controls, dead- 
zone, hysterisis, etc. Digital simulation of such functions, 
especially when employing fixed integration frame rates as 
normally used for real-time simulation, can result in very 
sizable dynamic errors unless very small integration steps 
are utilized. 

To summarize, real-time digital simulation of continuous 
dynamic systems, especially flight vehicles, is characterized 
by many input-output channels with high data rates, numer¬ 
ous multivariable functions requiring large data bases and 
many high-speed repetitive-type calculations, coordinate 
transformations which also tend to be repetitive in nature, 
a very large spread in dynamic frequency, and many nonlin¬ 
earities with discontinuities. The computer described in this 
paper has been designed for efficient processing of simula¬ 
tions having these characteristics. 

NUMERICAL CONSIDERATIONS 

An important parameter affecting both cost and perform¬ 
ance in the design of any digital processor is the word size 
used for arithmetic operations and memory. As stated in the 
previous section, much of the computational load in dynamic 
simulation of continuous systems, especially flight vehicles, 
involves evaluation of multivariable functions. Usually the 
data involved in the function tables is known with only 
limited precision, so that 16 bits is more than an adequate 


word size to represent the data points. Furthermore, with 
careful scaling techniques the evaluation of functions using 
table lookup and linear interpolation can be performed with 
a 16 bit arithmetic processor without any deterioration from 
16 bit precision. 1 Under these circumstances it would in fact 
be inefficient to use a word size significantly larger than 16 
bits. 

In the case of coordinate transformations the scaling of 
the required trigonometric functions is such that 16 bit pre¬ 
cision can be maintained. Additional algebraic computations 
are also required in general to compute the time derivatives 
of the state variables in typical simulations, but in most 
cases these do not include small differences of large quan¬ 
tities which might reduce the computational precision. Di¬ 
vision can be accomplished by generation of a reciprocal 
function followed by a multiplication. Simple scaling algo¬ 
rithms insure that precision is maintained even when the 
divisor varies over a large range. Expanded use of function 
generation instead of algebraic computation in many other 
formulas further improves the scaling. 2 

Thus for an accuracy range considered more than ade¬ 
quate for continuous system simulation, e.g., 0.01 percent 
to 0.1 percent, a convincing case can be made for using a 
fixed point 16 bit computation for the state-variable deriv¬ 
atives. Not only is the 16 bit precision adequate, but also 
the maximum range of state-variables is either known or can 
itself be computed, so that efficient scaling to take full ad¬ 
vantage of the 16 bit precision is straightforward. The use 
of true roundoff as opposed to roundup or rounddown in the 
arithmetic processor prevents bias which might cause error 
buildup. 

However, in the arithmetic involved in the mechanization 
of numerical integration formulas it is easily seen that 16 
bits can represent quite inadequate precision. In particular 
this is true when a simulation exhibits a very wide range in 
dynamic frequencies, which can often occur in flight simu¬ 
lation and other examples as noted in the previous section. 
The problem can be understood by considering a simple 
integration formula. Assume we wish to solve the equation 

dx 

-jj=Kf(x,y,. . .) (3.1) 

where x,y,. . . are state variables in the simulation. The 
constant K is chosen so that the function /ranges up to 
unity in magnitude. For simple Euler integration we use the 
following formula: 

Xn+i=Xn + hKf(x n ,y n ,. . .) (3.2) 

where h is the numerical integration step size and x n =x(nh), 
yn = y{nh), etc. Equations similar to (3.2) will be repeated 
for each of the problem state variables. As a result there 
will be a maximum scaling constant, K max , associated with 
one of the equations and a minimum scaling constant, ^ min , 
associated with another of the equations. In the flight-equa¬ 
tion example of the previous section, K max /K min = 25/ 
.01=2500. To be conservative, assume 

f a ^=10 4 . (3.3) 
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Now the maximum allowable numerical integration step 
size, h, will depend on K max . In particular, the modal time 
constant associated with the equation x=K max f=K max x, 
assuming f(x,y ,. . .) can be represented quasilinearly by the 
simple function x, will be l/K max . Let’s further assume that 
we take 10 3 integration steps per time constant, 1 /K max . This 
would yield good dynamic accuracy (~0.1 percent) using 
Euler integration. To be sure, higher order integration for¬ 
mulas would allow much larger step sizes for comparable 
accuracy. However, even in that case one would like to 
preserve the option of using the smaller step size without 
getting into roundoff problems. Thus we assume that 

h= xlO- 3 . (3.4) 

^max 

Now let us examine Eq. (3.2) for the case where K=K min . 
Then we can write 

Xn +1 Xn~^~ h^min f (3.5) 

or from Eqs. (3.3) and (3.4) 

x n +i—x n + xlO -3 / (3.6) 

•''■max 

=x n + l0~ 7 f~x n + /x 2 -23 . (3.7) 

If the function /is computed to 16 bit precision, and we 
wish the increment /x2 -23 to be added to x n with 16 bit 
precision, then clearly x n must be represented to at least 
23+16=39 bits of precision. Certainly 32 bits would be in¬ 
adequate and 48 bits contains a conservative safety factor. 
This is the rationale behind using 48 bits for the numerical 
integration processor even though only 16 bits is used in the 
computation of the derivative function /. The above argu¬ 
ment also explains why many 32 bit floating-point compu¬ 
tations which effectively realize only 21 to 24 bits of man¬ 
tissa precision must actually be mechanized in double 
precision for simulations involving “stiff’ systems (large 
ratios for K max / K mln ). 

In order to confirm the effectiveness of 16 bits for the 
computation of state variable derivatives and 48 bits for 
numerical integration, a number of typical systems have 
been simulated on a general purpose computer with a pro¬ 
gram using the word lengths as proposed above, along with 
true roundoff, These simulations, including nonlinear air¬ 
frame dynamic equations, have demonstrated the validity of 
the approach. 

TECHNOLOGY CONSIDERATIONS 

As we have seen, many continuous system simulation 
applications impose severe requirements in speed, real-time 
operation and economics. We also have observed that 16 bit 
fixed point computation is adequate in many important sim¬ 
ulation applications. A significant exception is implementa¬ 
tion of integration algorithms, where a 48 bit word length is 
desirable to accumulate small increments accurately. Re¬ 
flecting this application insight, a specialized digital com¬ 
puter for continuous system simulation should maximize, in 


some reasonable sense, its performance to cost ratio by 
matching real-time speed and numerical requirements to the 
currently available technology. This kind of optimization is 
reflected in the choice of memory, the logic family, the 
system architecture and a software approach. 

Factors in this kind of optimization include the following: 

(1) large data memory for function data and solution stor¬ 
age (up to 256K words or more), 

(2) high data rates to and from data memory, 

(3) fast 16 bit addition and multiplication rates for irreg¬ 
ularly structured computations, 

(4) flexible and fast accumulation of 48 bit operands, 

(5) functional parallelism and pipelining of different as¬ 
pects of the computational tasks, 

(6) architectural flexibility to grow capability with mini¬ 
mal interaction on existing hardware and software, 
and 

(7) very high speed analog to digital, digital to analog and 
digital to digital data transfers. 

Requirements (1) and (2), at first glance, are incompatible 
in a cost to performance optimization, since low cost mem¬ 
ory technology (cost per bit) emphasizes moderate speeds. 
Thus the currently available 4Kx 1 and 16Kx 1 MOS RAMS 
provide the lowest cost memory but with cycle times from 
200 to 500 nanoseconds. This compares with the much faster 
IK bipolar RAMS which offer 50 nanosecond cycle time but 
have much higher cost, increased power requirements and 
less packaging density. This dilemma is avoided by using 
the slower MOS RAMS in a highly interleaved configura¬ 
tion, This strategy requires data organization that allows 
overlapped accesses. Since the continuous system simula¬ 
tion application 1 permits this kind of data organization, the 
architecture to be described uses an inter-leaved memory to 
realize both economic and speed objectives. 

Requirements (3) and (4) can be realized by parallelism or 
by very fast addition and multiplication hardware. One ex¬ 
treme approach to parallelism would be to use hundreds of 
inherently slow microprocessors, each implementing addi¬ 
tion and multiplication routines. This scheme imposes seri¬ 
ous architectural constraints on data management and hard¬ 
ware interconnection. An alternative is to employ the very 
high speed-to-cost performance of emitter coupled logic 
(ECL). With ECL technology, a 17 bit by 17 bit product can 
be obtained in less than 50 nanoseconds. Also, the ECL 
family permits very high speed data rates between registers 
and between blocks of processing hardware. For these rea¬ 
sons, ECL was selected. 

Requirement (5) just emphasizes the fact that memory and 
arithmetic speeds have practical limits and, at some point, 
additional speed improvements must come from parallelism 
and pipelining. These techniques can give striking perform¬ 
ance improvements, but with penalties in increased cost, 
increased hardware complexity, difficulty in data transmis¬ 
sion, and severe programming and software constraints. 
These penalties become overbearing in the design of a gen¬ 
eral purpose digital computer. However, for an application- 
centered digital computer, which limits scope, the penalties 
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are more easily controlled and parallelism and pipelining 
offer intriguing possibilities. 

In particular, we can realize significant performance ad¬ 
vantages by designing each processor for a given function. 
Each processor of the multiprocessor computer is then dif¬ 
ferent and suited to a class of operations. An important 
fringe benefit of this approach is ease in meeting the archi¬ 
tectural flexibility of Requirement (6). In fact, the numerical 
integration processor, which will be described, is being 
added to existing systems with no difficulty. 

Requirement (7) is faced in interfacing the simulation com¬ 
puter to real-time hardware. To be compatible with a high 
speed digital architecture, both the analog to digital and 
digital to analog converters must be parallel. Fortunately, 
fully parallel analog to digital conversion is now feasible 
because of the improved performance and decreased cost of 
modular analog to digital converters. Digital to digital trans¬ 
fer is now encountered frequently as digital computers be¬ 
come an integral part of real-time hardware. The net effect 
of these real-time data conversion and transfer requirements 
is a special interfacing approach to the digital computer 
hardware. This implies simple, fast, burst-oriented transfer 
of data blocks to and from the data conversion equipment. 


COMPUTER ARCHITECTURE OF THE AD-10 

Figure 1 is a block diagram showing the architectural 
features of the special digital computer suited to the contin¬ 
uous system simulation task. This system, designated the 
AD-10 and manufactured by Applied Dynamics Interna¬ 
tional, will be described in the context of the major system 
elements shown in Figure 1. The purpose is to show how an 
application-architecture viewpoint can lead to truly signifi¬ 
cant performance improvements. 

Data, address, control and status multibus 

The data, address, control and status multibus is a parallel 
ECL bus composed of 16 data lines, 20 address lines and 12 
control and status lines. This high speed multibus supports 
20 data/address bus transactions per microsecond. These 
bus transactions pass data to and from data memory, five 
functional processors, the real-time input-output channel 
and the host digital computer. 

The transactions, as well as all memory and processor 
functions, are synchronously controlled from a master 40 
MHz clock. This synchronous approach maximizes speed 
by avoiding handshaking delays characteristic of asynchron¬ 
ous control. 

Host interface control 

All functions of the system are under control of a host 
computer interfaced to the multibus through the host inter¬ 
face control (HIC). Through the HIC, the host computer 
loads the program memory resident in each functional pro¬ 


cessor, loads data into the interleaved data memory, super¬ 
vises run/halt control of the system, and performs other 
control, status interrupt and diagnostic functions. The host 
may be a relatively large or a moderately sized Digital 
Equipment PDP-11 system. In addition to control functions, 
the host provides software support for program generation, 
translators, file management and debugging and diagnostic 
aids. 

Except for these set up and control functions, real-time 
continuous system simulation is achieved through coordi¬ 
nated programs stored in the five functional processors. 
These programs access required data from data memory and 
control the input-output channel. The data, address, control 
and status multibus, under control of these processors, 
serve as the data communication path between the proces¬ 
sors and data memory. 

Interleaved data memory 

The interleaved data memory (DM) holds the large data 
tables for multivariate function generation, solution storage 
and general data storage. Data is organized in DM so that 
data bandwidth is at rates up to the 20 MHz bus transaction 
limit. With up to 64 interleaved 4K pages of 16 bit words 
(plus parity), full memory bandwidth can be realized in many 
applications. Addresses for data are generated by the mem¬ 
ory address processor (MAP). Memory size is expandable 
to 1024K words by using 16K RAMS instead of 4K RAMS. 

Functional processors 

The functional processors, COP (control processor), MAP 
(memory address processor), DEP (decision processor), 
ARP (arithmetic processor) and NIP (numerical integration 
processor), work in an overlapped and coordinated manner 
on different aspects of the simulation task. All these pro¬ 
cessors have certain common features, which will be dis¬ 
cussed. 

The functional processors synchronously execute instruc¬ 
tions at a 10 MHz rate. Each processor contains its own 
program counter and program memory. These program 
memories, which are loaded through the HIC, are 50 nan¬ 
osecond, 1 K word, bipolar writable control stores. The 
word length in each processor is matched to the needs of 
that processor. In total, these word lengths add to 272 bits. 
These processor instruction words, fetched with look ahead, 
execute in 100 nanoseconds and are responsible for the 
parallel computing power and speed of the system. 

The advantages of this distributed processor architecture 
are significant. First, the relatively independent character of 
each processor minimizes design and construction problems 
implicit in 100 nanosecond execution times. Second, the 
structure of each processor is determined by a few functional 
requirements. This is a great simplification over the complex 
job of designing one processor to do all things. Third, the 
flexible parallel structure allows the system to be tailored to 
different application without important impact on other ele- 
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Figure 1—AD-10 block diagram 


ments of the system. Fourth, the sequentially coordinated 
operation of the different processors minimizes program¬ 
ming difficulties inherent in parallel processing. This se¬ 
quential orientation is consistent with usual programming 
concepts and avoids the very difficult task assignment and 
data communication problems which characterize other par¬ 
allel architectures. 

The different functional processors and their roles in con¬ 
tinuous system simulation will be considered now in more 
detail. 

Control processor (COP) 

The COP coordinates the activities of the other functional 
processors and provides the means to request service from 
the host computer. In particular, COP instructions 

(1) provide halt and conditional halt control, 

(2) control the active/wait status of all processors, 

(3) load processor program counters and status words, 

(4) read status and data bits for conditional halts and 
jumps, and 

(5) control the real-time input-output channel. 
Programmed halts from the COP allow the host computer to 


control both single frame and periodic operation. Condi¬ 
tional halts permit error conditions or program conditions to 
halt the program and request host service. 

Each processor has an active and wait state. In the active 
state the program counter is counting and executing instruc¬ 
tions; in the wait state a processor is not executing instruc¬ 
tions. Through the active/wait instruction, the COP can con¬ 
serve program memory in a temporarily unneeded 
processor. 

Through load program counter,jump and conditional jump 
instructions, the COP provides a minimal looping and sub¬ 
routine capability. Because of the pipelined character of the 
interleaved memory and, to a degree, the processors, loops 
should be relatively large for speed effectiveness. Philo¬ 
sophically, the architecture is most capable in executing 
blocks of branch-free code. Thus, as an example, a linearly 
coded binary search is used to determine input location 
between function breakpoints. This technique fully controls 
the search time and avoids looping. 

Memory address processor (MAP) 

The MAP generates physical addresses for the DM from 
MAP user addresses. The physical address consists of a 
word address (one location in an interleaved page of 4096 
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locations) and a page address. User addresses are converted 
to physical addresses by an alignment network which as¬ 
sures that overlapped memory accesses will not be in time 
conflict in a computational task. 

Decision processor (DEP) 

The DEP executes comparison and modification instruc¬ 
tions on a dual 128 word register file within the DEP. This 
structure efficiently implements a binary search in function 
generation, 1 as well as other decision oriented operations. 
Because these decision operations are isolated within the 


DEP and because the DEP is associated with the MAP, the 
ARP is simplified and addressing is made more efficient. 

Arithmetic processor (ARP) 

The ARP provides the very high speed 16 bit arithmetic 
needed for continuous system simulation. Through over¬ 
lapped move and arithmetic operations and a very high 
speed 128 word register file, the ARP efficiently implements 
the function generation and irregularly structured compu¬ 
tations characterizing simulation applications. Arithmetic is 
in 2’s complement fractional and integer formats. Provisions 



Figure 2—Block diagram of the arithmetic processor 
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are made for full carrys, scaling and a true out-of-range 
saturation which avoids modulo 2 16 wrap-around. These fea¬ 
tures preserve the effectiveness of the 16 bit format. 

The organization of the ARP is shown in Figure 2. Four 
move subinstructions per instruction set up the operands for 
the following instruction. This following instruction executes 
an arithmetic subinstruction of the general form 
R=±(A±B)*C±D on the operands A, B, C and D, pro¬ 
ducing the result in 175 nanoseconds. Since all these func¬ 
tions are fully overlapped in a pipelined approach, the result 
R is computed with a 10 MHz throughput rate. Thus 20 
additions and 10 multiplications can be achieved each mi¬ 
crosecond. 


Numerical integration processor (NIP) 

As we outlined earlier, the right hand side of the simula¬ 
tion differential equations is adequately represented by 16 
bit arithmetic. The features in the processors described 
above are effective in implementing these right hand side 
computations. Many AD-10 systems are currently working 
in the field performing some or all of these operations. In 
these situations, the remaining computations and integra¬ 
tions are accomplished in the analog computer subsystem of 


a hybrid computer system or, in a few cases, in a general 
purpose digital computer. 

The full performance capabilities of the AD-10 for dy¬ 
namic system simulation are realized when all computations, 
including integrations, are done within the AD-10. This re¬ 
quires the numerical integration processor (NIP), which is 
shown in the simplified block diagram of Figure 3. The NIP 
elements include a 48 bit arithmetic unit, a parallel shifter 
for accumulating increments to 48 bit words, a 48x1 K 
register file, buffer stacks and data and control stack sub¬ 
processors. The NIP communicates to the other processors 
through the multibus and is controlled by microprogrammed 
instructions stored in the 80x 1024 program memory. 

Data, control and exponent stacks, and indexed address¬ 
ing to the register file are essential to subroutine implemen¬ 
tation of integration algorithms and the effective overlap of 
integration computation with computation occurring in the 
other processors. The parallel shift network positions a 16 
bit variable according to the sum of four shift factors, E0, 
El, E2 and E3. This efficiently implements integration al¬ 
gorithms and allows a “quasi” floating point format for 
difficult scaling situations which can occur occasionally. The 
control stack processor permits simple block transfers to 
other processors and efficiently controls substep logic in 
Runga-Kutta and other integration formulas. The data stack 



Figure 3—Block diagram of the numerical integration processor 
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processor is useful for variable step-size algorithms and for 
implementation of on-off functions, hysteresis, saturation 
and other simple but common nonlinearities. All these fea¬ 
tures minimize the interaction of the other processors, max¬ 
imize overall computing speed and overlap NIP functions 
with other processor tasks. The general effect is that inte¬ 
gration has minimal impact on frame time, which is therefore 
primarily determined by the other processors. 

The instruction fields of the 80 bit NIP instruction are 
relatively uncoded so NIP programs are microcoded in char¬ 
acter. A variety of subprograms are available for imple¬ 
menting a family of integration routines and other functions 
required in real-time continuous system simulation. The in¬ 
terfacing of these routines to the other processors is very 
simple, primarily involving data transfers from and to the 
multibus. 

The integration routines are all resident in the NIP so that 
integration formulas and step size can be changed at run 
time without any translation requirements in the host. This 
means that formula selection and step size optimization is 
a simple process. 

SOFTWARE 

Since the architecture of the AD-10 is specialized, it is not 
possible or practical to develop resident software. Instead 
the host computer and its software systems are used for all 
program development and interactive control of the AD-10. 
The Digital Equipment PDP-11 series, operating under RSX- 
11M, has up to now served in all cases as the host computer 
for the AD-10, although other host computers can be used. 

Available AD-10 software includes interface communica¬ 
tion routines, an interactive executive, a cross assembler, 
a macrofile library, a simulator, and hardware diagnostics. 
In addition, a high level interactive simulation language is 
under development. 

The interface communication routines are a set of subrou¬ 
tines which perform all the interface control functions be¬ 
tween the host PDP-11 and the AD-10. These routines are 
callable from either Fortran or assembly language programs. 
They are used by the interactive executive and are also 
available for user application programs. 

The interactive executive (ADX) is an interactive program 
for loading, running and debugging AD-10 programs. ADX 
accepts commands from a user at a terminal or from a 
command file. ADX allows the user to start and stop the 
AD-10, step its program through specified instruction cycles, 
and display and modify registers, data memory and program 
memory. The file structure of RSX-11 is available to ADX 
to load files created by the cross assembler or to save spec¬ 
ified sections of AD-10 data and program memory. 

The AD-10 cross assembler generates object modules 
which are loadable by ADX. The assembler produces a 
comprehensive program listing, which includes the object 
code generated for each source statement, a symbol cross 
reference table and a detailed timing analysis of all multibus 
transactions which occur during program execution. The 
timing analysis and multiprocessor features of the assembler 


simplify the problems of relating the code associated with 
each processor. Conventional procedural reasoning is still 
valid in writing programs, even though this might not be 
expected because of the multiple processors. 

User programming, at the assembly level, is simplified by 
an extensive library of application macrofiles. A macrofile 
is an AD-10 assembly language routine, in source file form, 
which can be included in a user application program by 
specifying input/output parameters or arguments. These rou¬ 
tines, optimized for speed and accuracy, account for the 
great majority of code in most applications. 

After assembly a program can be fully simulated in the 
host computer by the AD-10 simulator. This permits full 
debugging and program evaluation within the host. The sim¬ 
ulator uses ADX for its control so the normal operating 
environment is preserved in the use of the simulator. 

A high level simulation language for the AD-10 is now 
under development. This language, while having a few con¬ 
straints not found in other continuous system simulation 
languages, will allow users to take full advantage of the AD- 
10 speed with minimum programming effort. The system 
will emphasize interactive involvement with the model and 
graphic display of solutions. It will be possible to optimize 
integration formulas, change scaling, change model param¬ 
eters and initial conditions, document models and studies, 
and prepare hardcopy graphics. The AD-10 simulation lan¬ 
guage will also be compatible with a host simulation lan¬ 
guage so that the host computer can be used to generate test 
solutions and scaling information. 

EXAMPLE 

To illustrate the computational speed of the AD-10 mul¬ 
tiprocessor described in this paper, the equations necessary 
to simulate an aircraft in six degrees of freedom were con¬ 
sidered, including extensive aerodynamic functions. 3 One 
evaluation of all twelve state variables derivatives requires 
47 adds, 108 multiplies, and 11 trigonometric functions in 
addition to computation of 2 two-variable, 8 three-variable, 
and 7 four-variable functions. Using the Macrofile Library 
routines for function generation, coordinate transformations 
and data transfers, as currently available in the standard 
AD-10 software, the time required for a complete pass 
through all the equations is less than 100 microseconds. 

If an extrapolative type of numerical integration routine 
is used, then only one pass through the equations is required 
per numerical integration step. Since operations in the NIP 
(numerical integration processor) go on concurrently with 
those in the ARP (arithmetic processor) it is reasonable to 
assume that numerical integration of the 12 state variables 
will not add significantly to the frame time. Thus we will 
assume 100 microseconds per numerical integration step. 
Using third-order Adams-Bushforth (a popular integration 
routine for real-time digital simulation), roughly 50 integra¬ 
tion frames per cycle of the highest problem frequency are 
needed for dynamic accuracy of the order of 0.1 percent. 
Thus the period of the highest problem frequency would be 
50 (100) or 5000 microseconds, which corresponds to 200 




Design Considerations in a Multiprocessor Computer 


393 


hertz. Since the highest frequency associated with the rigid- 
body equations of a typical high performance aircraft is 
about 1 hertz (the so-called short-period pitching mode) the 
conclusion is that the AD-10 can solve the flight equations 
at 200 times real-time. 

If the aircraft simulation also includes the flight control 
system, and perhaps elastic structural modes, then frequen¬ 
cies up to 25 hertz and higher may be present in the problem. 
Even allowing for the increased equation complexity in this 
case, it is clear that the AD-10 should be able to handle the 
real-time simulation comfortably . 

CONCLUSIONS 

Design of the AD-10 multiprocessor, optimized for contin¬ 
uous system simulation, has been described. The computer 
is characterized by a large high-speed interleaved memory, 
very fast arithmetic processor speeds, many high-speed 
input-output channels, and operation under a host general- 
purpose digital computer. The AD-10 multiprocessor ap¬ 
pears capable of handling the most demanding of real-time 


aerospace vehicle simulations. Computing speed is approx¬ 
imately two-orders of magnitude faster than the speed of 
comparably priced general purpose minicomputers in sim¬ 
ulating continuous systems. Six-degree-of-freedom aircraft 
flight equations can be solved at up to 200 times real-time. 
Such faster-than-real-time solutions could be very cost ef¬ 
fective in parameter optimization problems and problems 
involving stochastic inputs, where large numbers of repeti¬ 
tive solutions are required. 
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INTRODUCTION 

Computer simulations have played an important role in fu¬ 
sion energy research. Although the basic physical principles 
of plasma physics are well understood, the coupled physical 
phenomena are so complex that analytical analyses cannot 
be easily carried out. Moreover, laboratory experiments are 
costly. With the enormous power of modern computers, one 
naturally resorts to computer simulations to gain insights 
into physical phenomena, as well as to guide the design and 
planning of experiments. 

In order to cope with the rich and disparate phenomena 
occurring in a plasma, a wide variety of simulation models 
are required. 1 The models may generally be classified as: 
particle models, which treat the plasma at the microscopic 
level by following the motion of a large number of particles; 
fluid models, which treat plasma on a more macroscopic 
level; and also hybrid models which are a mixture of the 
first two models. 

It is obvious that one cannot do plasma experiments in 
a computer with the 10 12 to 10 19 particles involved in physical 
problems of interest. Still, one would hope to have as many 
as 1 million particles on a mesh system of 64 3 for particle 
codes. For fluid codes, one may want a system of 64 3 to 
have better resolution, especially when local structure is 
present. 

This shows that plasma simulations require a large com¬ 
puting capability and therefore have only been carried out 
on large computers, such as the IBM360/91 or CDC7600. 
Nevertheless, few attempts have been made for systems as 
large as mentioned above, for they need several megawords 
of storage and a considerable amount of computer time for 
a significant run. 

Lately, with the development of computer technology, 
microprogrammable specialized minicomputers appear to 
provide the possibility of doing some of the jobs effectively 
and economically. Because of a successful pilot study 2 ini¬ 
tiated by John M. Dawson and Burton D. Fried of the UCLA 
Plasma Physics Group and Glen J. Culler of Culler/Harrison, 
Inc. (CHI), a CHI plasma simulation computer system is 
being installed at UCLA. Our preliminary results indicate 
that the system offers a significant improvement in perform¬ 
ance and, most importantly, it reduces cost by about two 
orders of magnitude from the IBM360/91. 


The CHI plasma simulation system consists of six micro¬ 
processors: one array processor (AP-120B, manufactured by 
Floating Point Systems, Inc.), 3 one macro-processor (MP-32 
made by CHI) 4 and four I/O processors (IOP made by CHI). 
The AP does most of the calculations, the IOP moves data 
around, and the MP does scheduling and control. Mass data 
are stored on disk. The parallel processing among and within 
these processors accounts for most of the performance ad¬ 
vantage of the system. Further details of the system are 
given in the third section of this paper. 

Three plasma simulation codes have been tested on the 
system: a 2 -Vi dimensional electrostatic particle simulation 
code, 2 and both 2 -Vi and 3 dimensional ideal MHD fluid 
codes. 5 The basic structure of these codes is described in 
the third section, together with comparative performance 
data. Conclusions are presented in last section. 

THE UCLA CHI COMPUTER SYSTEM 
General description of the system 

The configuration of the UCLA CHI computer system is 
shown in Figure 1. It includes a macro-processor (MP-32), 
an array processor (AP-120B) and four input/output proces¬ 
sors (IOP). Data Throughput and capacity of each processor 
is summarized in Table I. The AP does high speed floating 
point arithmetic operations. Further descriptions of the AP 
are given later. 

The MP is used for control, I/O, bookkeeping calculations, 
and integer arithmetic operations, and serves as the host 
computer for the AP. In the present system it supports an 
interactive mathematical system (Math System), 7 provides 
a text editor, file facilities, the necessary assemblers and 
linkers. 

The IOP’s are fixed program micro-processors with access 
to both AP and MP main data memories and the UNIBUS. 
Their primary function is to provide high level control over 
access to the disk drives. Each IOP functions as a disk 
formatter supporting any of the four disk drives. This allows 
the configuration of individual drive capacities from 5 million 
to over 60 million 38 bit words. Transfer time per word is 
under 4 /u,s. In addition to transferring data between the AP 
main data memory and disk, the IOP can also transfer data 
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TABLE I.—Data Throughput and Capacity of the AP-120B, the MP-32A, 
the IOP and disks of the CHI System 

Macro Processor (MP-32A) 

(a) A 16-bit fixed point processor with a 1/6 ys machine cycle. 

(b) Memories: 

64 words scratch pad data memory 
512 words fixed instruction memory 
64 words writeable instruction memory 

65,536 words of 1/3 ys instruction and data memory (CD). 

(c) Integer multiplication of two 16-bit words in 1/3 ys. 

(d) * Functions: 

(i) Controls high-speed local station displays. 

Display rate: 2 ys/point 

< 2 ms for longest line 
> 4000 characters/second. 

(ii) Schedules IOPs and Array Processor. 

(iii) Controls data transfer to modems (1200-9600 baud) and 
other PDP-11 compatible I/O devices over Universal I/O bus. 

Disk-IOPs 

(a) Disk capacity: 4 Trident T80 drives - over 16 million 38-bit words each. 

(b) IOPs are fixed program micro-processors. 

(c) Data transfer rate: Data on all four disks can be simultaneously 
transferred to/from AP-MD or MP-CD, each at 250,000 words/second. 

Any IOP can be used for memory transfer between AP-MD and MP-CD at 
1,000,000 words/second. 

Array Processor (AP-120B) 

(a) A 38-bit floating point arithmetic unit with maximum speed of 12 megaflops. 

(b) Memories: 

(i) Two 32-word data pad memories 

2560 words 1/3 ys fixed table memory 
65536 words 1/3 ys data memory (MD) 

512 words instruction memory 

(ii) Data pad access is immediate. Table memory may be referenced 
every cycle, with an access delay of two cycles. Data memory 
may be referenced every other cycle, with an access delay of 
three cycles. 
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Figure 1—Block diagram of the UCLA CHI computer system 


between the MP memory and disk or perform memory to 
memory transfers between or within either memory. 

Each processor can process independently while the 
whole system is controlled by the MP (host processor). Each 
processor is scheduled to perform a specific process. It 
interrupts the host when it completes each process, then 


begins immediately, without having to wait for service from 
the MP, on its next process if one has been scheduled for 
it. This parallel logic computing capability is one of the main 
features of the system and makes possible the handling of 
large scale computing problems. An illustration is given in 
the third section. 
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AP-120B processor 

Since all floating point calculations (therefore most of 
model calculations) are performed in the AP, it is of impor¬ 
tance to explore the structure of the AP. 3 The architecture 
of the machine is shown in Figure 2; and its features are 
summarized in Table I. 

Special features are noted as follows: (1) Programs, con¬ 
stants, and data each reside in separate, independent mem¬ 


ories: program source memory, table memory and main data 
memory, respectively. Thus, it eliminates memory accessing 
conflicts. (2) Independent pipelining floating-point multiplier 
and adder units allow multiply and add operations to each 
be initiated every Ve p,s. (3) Two blocks of fast access 
floating-point accumulators, DPX and DPY with 32 locations 
each, are available for temporary storage of intermediate 
results from the multiplier, adder, or memory. (4) Address 
indexing and counting functions are performed by an inde- 



Figure 2—Block diagram of the AP-120B micro-processor (From Reference 

3) 
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Figure 3—AP-120B Instruction Field Layout (From Reference 3) 


pendent integer arithmetic unit (S-Pad and ALU). (5) Float¬ 
ing-point data is represented by 38-bit words which gives a 
precision of 8.1 decimal digits. 

Each instruction word has 64 bits so that many different 
operations may be performed concurrently during each 
cycle. The instruction field layout is shown in Figure 3. It 
is separated into five general areas: program control, address 
control, arithmetic, memory and I/O. The highly parallel and 
pipeline structure of the AP-120B make it possible to achieve 
fast execution with lower speed logic and thus with lower 
cost. It requires x h /as for a single multiplication and x h /as 
for a single addition in the AP. But with adder and multiplier 
units working in parallel and with their pipeline structure, 
the AP allows multiply and add operations to each be initi¬ 
ated every Ve /as. This gives a maximum speed of 12 meg¬ 
aflops comparable with a large general purpose computer. 
In the three test plasma simulation problems the speed 
achieved is about 4 megaflops for the fluid codes and 6 
megaflops for the particle code. 

In order to achieve this high speed capability, each in¬ 
struction of a micro-coded program should use as many 
fields as possible. For this reason, our experience in coding 
plasma simulation codes suggests that calculations done in 
the manner of particle by particle, or grid-point by grid-point 
may be more efficient than trying to vectorize the code. 
Besides, vectorization usually requires several additional 
arrays for intermediate results. In contrast, vector opera¬ 
tions are useful in applications such as digital signal proc¬ 
essing, where the AP is called upon to perform vector op¬ 
erations at high speed. The Math System is designed for 
efficient support of these AP vector operations. 

The basic arithmetic operations in the AP are addition and 


multiplication. The divide operation is carried out in soft¬ 
ware with speed of 3.83 /as. Hence, it is essential in the 
programming to use as few divide operations as possible. 
For instance, there is no divide operation in the 2 V 2 D par¬ 
ticle code; however, one such operation is required in the 
MHD codes. In the 2 V 2 D MHD code, the divide operation 
uses 7 percent of the total processing time of 50 /as per grid 
point. 

As previously mentioned, in the CHI system, the IOP’s 
have a separate data bus into the AP main memory from 
that used by the AP processor. Thus the CHI system elim¬ 
inates the bottleneck problem 8 which may exist in other host 
setups where all data must enter and leave the AP through 
the interface module and the host computer. 

Programming languages 

Programming can be carried out in four languages, which 
include two machine languages, the AP- and the MP- micro 
languages; and two high level languages, the MP-macro lan¬ 
guage 6 and the Math System language. 7 The AP micro-in¬ 
structions have just been mentioned. The MP micro-instruc¬ 
tion consists of 28 bits and provides for parallel operations. 
The MP macro-language is a set of macro-instructions which 
are pre-programmed in both the AP and the MP micro-lan¬ 
guages and is used for system programming. 

The Math System language, programmed in the MP micro- 
and macro-language is an interactive interpreter, and sup¬ 
ports operations of mathematical functions applied to real 
or complex vector data as well as useful graphics. It can 
execute either interactively or from a program consisting of 
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© A , B -*• C 


"C(i): = A(i) + B(i) for all i 



"C(i): = sin C(i) for all i 


( DISPLA^ ) "plot C(i) vs. i 

Figure 4 —A Math System program for the calculation, Ci=sin(Aj+B(), 
with three vectors A, B and C 


these mathematical operations. Since the Math System uses 
vector data which are entities usually occurring in physical 
problems, programming in this language is more concise and 
less error-prone than in the standard Fortran language. 

A Math System program is shown in Figure 4 for the 
calculation of Ci =sin(A i + B { ) with three vectors A, B and 
C. The operation © A,calculates C i =A i +B i for all i, 
so does the operation SIN C for Cj=sin (Ci). The program 
therefore saves an explicit loop that is required in a Fortran 
program. The statement DISPLAY will plot C t against i in 
terms of its natural units. 

In plasma simulation codes, we anticipate using only the 
Math System language and the AP micro-language. AP 
micro-programming is done only where the development of 
maximally efficient routines is warranted, as the particle 
push routines for particle simulations. The remaining rou¬ 
tines, such as initialization, control, and force-calculation 
routines in the particle codes, which require only a small 
fraction of total processing time, are easily coded in the 
Math System language. 

The operating system supports both batch and on-line 
usage. Thus, it allows old production or test runs of a code 
simultaneous with code development, editing or on-line 
Math System usage. 


PLASMA SIMULATION MODELS AND THEIR TEST 

RESULTS ON THE CHI SYSTEM 

A particle code and a fluid code, which together represent 
a wide spectrum of plasma simulation models, have been 
adapted to the CHI computer system. The particle code is 
the standard 2 x k dimensional electrostatic model with a 
constant magnetic field plus mirroring. 2 The fluid code is an 
ideal magnetohydrodynamics (MHD) code, 5 which is being 
implemented in both 2 l k and 3-dimensional versions. These 
are very basic models that include the most important gen¬ 
eral features of plasma simulations. 

In this section, we present basic structures of these codes, 
the utilization of the resources of the CHI system and their 
test results. Details of the models and their numerical 
method aspects will not be discussed here. 


A 2 1 /2 dimensional electrostatic model with a constant 

magnetic field 

In a particle model, the trajectories of a large number of 
charged particles (electrons and ions) under the influence of 
self-consistent fields are followed in time. The particular 
model considered consists of a large number of charged rods 
parallel to the z-axis which are allowed to have velocity 
components in the x, y, and z directions as shown in Figure 
5. It allows spatial variation in the x and y directions. The 
system is periodic in both x and y directions such that par¬ 
ticles that leave one side of the system are re-entered at the 
opposite side. A uniform static magnetic field is permitted 
to point in an arbitrary direction. The charged rods move 
under the influence of their self-consistent electric fields and 
the given magnetic field. In addition, the rods may be re¬ 
flected at the y -boundaries if a mirroring condition is satis¬ 
fied by the velocity components parallel and perpendicular 
to the magnetic field. 

The motion of the system is generated in the computer by 
a leap-frog time difference scheme, using a standard finite- 
size particle model, in which the charge of a particle is 
distributed over a finite region about the center of mass, and 
also using a dipole expansion technique. The finite-size par¬ 
ticle model effectively reduces undesirable collisions and 
also numerical noise from the discrete nature of the spatial 
grid. However, in the following discussions, only the basics 
of the model are presented. 

Mathematically, the simulation model solves the following 
equations self-consistently: 

V-£(r)=p(r), (1) 

(£('<)+ iixiHr,)), ( 2 ) 

dt mi 

dh . 

it =i ' (3) 

E and B are the electric and magnetic fields, respectively; 

B is assumed to be a constant vector, p is the charge density 
distribution and r t , v t stand for the position and velocity of 
the z'th particle with charge q t and mass m t . 

In the 2 V 2 D model, a spatial grid mesh in the x-y plane 
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Initialize: 


(1) Random Numbers 

(2) Disk Variables 

(3) Control Parameters 


(1) Read input and generate constants. 

(2) Compute initial position, velocity 
for every particle. 

(3) Compute initial charge distribution. 


Compute forces on grid points from 
charge distribution using Fast Fourier 
Transform. 


IOP, AP Work in parallel to: 

(1) Update position, velocity of each 
particle. 

(2) Compute new charge distribution. 

(3) Compute energy for error monitoring. 


Computes one or more of: 

(A) Temperature 

(B) Diffusion 

(C) Flux-tube charge density. 


Figure 6—Program flow of the particle model 
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is introduced. All macroscopic quantities, such as E and p 
are calculated on the grid points. The basic flow of the 
program is shown in Figure 6. It starts with setting up initial 
configuration of the system by assigning v t and r t and q t for 
each particle and specifying the B field. 

Inside the loop, calculations are performed in two stages: 
(a) Force calculation: p(r) is used to solve Eq. (1) for the 
electric field E. Eq. (1) is solved efficiently by fast Fourier 
transforms. (This technique is especially useful when finite- 
size particle model is used.) (b) Particle push: After E is 
obtained, the particle positions and velocities are advanced 
in time by means of Eqs. (2) and (3), using the leap-frog 
method with a dipole expansion of the force. As each par¬ 
ticle position is updated, its contribution to p(r), including 
dipole terms, is calculated. We are now ready to start the 
next time step. 

In the actual code with the finite size particle model and 
dipole expansion technique, the data required are 6 words 
per particle, i.e., r and v with three components each, 9 
words per grid points which include E x , E y , p, D x , D y , E xx , 
E xy , E yx , and E vv . (The last six variables are dipole terms.) 
Since particles are grouped according to species, it is not 
necessary to specify charge and mass for each particle in¬ 
dividually. 

For a 10 6 particle and 64 2 grid-point system, more than 90 
percent of the computer time is used in the particle PUSH 
routine. Therefore the PUSH routine is coded in the AP 
micro-language with maximal optimization, whereas other 
routines are efficiently programmed in the Math System. 
The PUSH routine takes 13 /xs per particle which is four 
times the speed of the corresponding IBM360/91 assembly- 
language code. The AP PUSH routine is not a direct trans¬ 
lation of the corresponding Fortran code, but a new design 
for an efficient usage of the AP. There are 33 multiplications 
and 45 additions in the routine, and therefore an effective 
speed of 6 megaflops is achieved. 

The data storage requirement as well as processing timing 
are given in Table II. To handle the large quantity of data 
the technique of parallel processing among the AP, the MP 
and the IOP’s is used. To further explain this parallelism, a 
section of the Math System program is shown in Figure 7. 
The section is essentially the control program for pushing 
particles. The actual code, however, is a simple three-state¬ 
ment loop program for this two-disk system. 

The instruction, OP PUSH BUFFI causes the AP to ini¬ 
tiate or schedule an AP micro-program named PUSH for 
advancing particles whose descriptors are in the array 
BUFFI of AP main memory. The execution will begin as 
soon as the AP has completed all its previously scheduled 
processes and data BUFFI is free from operations associ¬ 
ated with other processors. If the AP is ready but data block 
BUFFI is still being loaded with data by the IOP, the AP 
will wait until the latter action is completed. The instruction, 
MOVE INDISK—»BUFF1, will instruct an IOP to move 
data from disk that is used for input to the location BUFFI 
in AP main memory. An IOP will perform the operation 
once it comes to the instruction and neither block of data is 
being used by other processors. Similar operation is defined 
for the instruction MOVE BUFFl-^OUTDISK. 


Math System Program 

Processor 

MOVE 

INDISK BUFF3 

I0P1 

MOVE 

BUFFI + OUTDISK 

IOP 2 

OP 

PUSH BUFF2 

AP 

MOVE 

INDISK BUFFI 

I0P1 

MOVE 

BUFF2 ■* OUTDISK 

IOP 2 

OP 

PUSH BUFF3 

AP 

MOVE 

INDISK BUFF2 

I0P1 

MOVE 

BUFF3 -* OUTDISK 

I0P2 

OP 

PUSH BUFFI 

AP 


Figure 7—A section of a Math System program for controlling the AP and 
the IOP’s in particle push 


Therefore, in executing the program shown in Figure 7, 
two IOP’s and the AP are operating simultaneously: One 
IOP loads into one block of AP main memory a block of old 
particle data from one disk, the AP executes PUSH routine 
in advancing particle data stored in the second block of AP 
memory, while the other IOP stores a third block of updated 
particle data from AP memory into the second disk. By 
choosing adequate blocksize and using enough number of 
IOP’s, it is possible to eliminate restrictions on over-all 
processing time imposed by I/O. 

To handle 10 6 particles, four disks are used with two disks 
for input, and two disks for output. The two pair of disks 
are interchanged with respect to I/O at each time step. The 
size of block is chosen to be 4K words (a track or 680 
particle descriptors). It takes 17.7 ms for the AP to update 
two blocks of data and 20 ms for each IOP to transfer one 
track of data between disk and AP memory. (The transfer 
rate of 20 ms per track includes a 20 percent allowance for 
the loss of one rotation at the cylinder boundary). The over¬ 
all processing time is then slightly limited by I/O to 14.7 ps 
per particle. 

Magnetohydrodynamic codes 

The set of magnetohydrodynamic equations includes: 

|£=-V-(p5)+£VV, (4) 

h ipv,)= ~£r l { p ‘’ ,Vl+ { pT+ t) s «~ b ‘ b >) 

+ vV 2 pVi-\pVi~ pg, (5) 

3 D 

— =Vx(vxB)+r,V 2 B, (6) 
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Data: 


TABLE II.—Resource Utilization and Processing Timing for a 10 8 Particle 
and 64 2 Grid Point Particle Simulation Code on the CHI system 


6 Words/particle x, y, z, v^, v , 

9 Words/gridpoint: E , E , E ,E , E , E ,C,D,D 
/& r x y xx xy yx yy x y 


The particle descriptors are on disk, 680 particles/track, 1472 tracks, 
or 736 tracks on each of two disks for input, and the same for output, 
with the two pairs of disks being interchanged with respect to I/O at 
each timestep. 


AP-Memory Allocation: 


Electric Field arrays 

(6): 

24k 

Charge and dipole arrays (3): 

12K 

Particle buffers (5): 


20k 


Total 

56k 


Data Transfer Timing : 

60 disk rotations/second or 16.7 ms/track, for a transfer rate of 
24.5 ps/particle for each of two disks, or an overall rate of 
12.2 ys/particle on a track-by-track basis. Loss of one rotation 
at the cylinder boundary (5 tracks/cylinder) increases this by 20%, 
giving 15 ys/particle on a cylinder-by-cylinder basis. 

Processing Time : 

Dominated by the particle-push AP routine for this system size. 
Present AP code estimated at 13 ys/particle. 

Thus the AP processing is slower than the I/O on a track-by-track 
basis, but is faster than the I/O on an overall basis. 


Overall Timing : 

15 ys/particle overall, or 15 seconds/timestep. 

16 hours for a run of 4000 timesteps (to u^t = 1000 @ At = 0.25/^) 
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Data: 


TABLE III.—Resource Utilization and Processing Timing for a 2-Vi D 
MHD Code 


7 variables: 


V ,V ,V , B , B , B ,p 

x y z x y* z 


Each variable occupies an array of dimension (N + 3)*(N^ +2), where 

N and N are the numbers of grids in x- and y-direction, respectively, 
x y 

The system is periodic in y and bounded in x. Temperature is assumed 
to be constant. 


AP-Memory Allocation : 

7*(N + 3)*(N + 2) + 30 words, 

x y 

With 45K memory, the system can be as large as 80 x 80. 


AP-Program Memory : 

445 program words. 


Processing Time : 

55 ysec/grid for a 20 x 20 system. 

50 ysec/grid for a 40 x 40 system. 

There are 1/2*(N + 1)*N grid calculations per time step in the 
x y 

leap-frog scheme. 


^- = -V-(Tv)+ (2- y)TV-v+ - (VxB) 2 

at p 



+ (y— 1)kV 2 T. (7) 

Here, the mass density is denoted by p, the velocity by v, 
the temperature by T, the magnetic field by B and the grav¬ 
itational acceleration by g. y is the ratio of specific heats. 
The transport coefficients e, v, X, t) and k are assumed to 
be constants. 

The computer code solves the equations as an initial-value 
problem, by integrating them in time using an Eulerian grid 


and an explicit, conservative, leap-frog finite-difference 
scheme. The details of the numerical procedures can be 
found elsewhere. 1,5 

All variables are macroscopic and are defined at grid 
points. In a three-dimensional code, a three dimensional grid 
is introduced. For the case of 2 l k dimensional code, a two 
dimensional grid is used, with v and B still having three 
components. 

The basic flow of the program as shown in Figure 8 is a 
loop over timestep, similar to that of particle model, except 
here the routine STEP performs one timestep integration of 
the MHD equations for all grid points. The routine STEP is 
coded in the AP micro-language, whereas other routines 
including initialization and control are in the Math System. 
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Initialize: 

(1) Control parameters 

(2) Disk variables 


(1) Compute initial plasma 

configuration-set up initial 
values for p, v, B and T 
or 

(2) If restart, read input from disks 


AP and IOPs (if any) work parallel to 
integrate MHD equations from time t 
to t + At over all grid points. 


Compute various energies, output 
p, v, B and T distributions. 


Figure 8—Program flow of the MHD fluid codes 
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TABLE IV.—Comparison between the CHI System and the IBM360/91 
with Respect to the Three Test Plasma Simulation Codes 



IBM360/91 

CHI SYSTEM 

PARTICLE CODE 



Time/Particle Push 

Assembly Code 


CPU 

55 ys 

13 ys 

I/O 

- - 

0.6 x 24 ys 

Overall 

55 ys 

14 ys 

10 ^ Particles, 4000 Timesteps 



Time 

61 hr 

16 hr 

CPU Cost 

32 k$ 

- - 

I/O Cost 

18 k$ 

- - 

Total Cost 

50 k$ 

0.3 k$ 

2-1/2D MHD CODE 

Fortran 

Code 

(H-Compiler) 


Time/Gridpoint 

130 ys 

50 ys 

3D MHD CODE 

Fortran 

(H-Compiler) 


Time/Gridpoint 

230 ys 

90 ys 

3 

64 System, 4000 Timesteps 



Time 

33 hr 

13 hr 

CPU Cost 

17 k$ 

- - 

I/O Cost 

10 k$ 

- - 

Total Cost 

27 k$ 

0.24 k$ 


The routine STEP is coded in such a way that calculations 
are done grid-point by grid-point, which seems to be the 
most efficient method in utilizing parallel operations in the 
AP. The processing time for the 2 V 2 D code is 50 /as per grid 
as compared with 130 /as for a corresponding Fortran code 
on IBM360/91. For the 3D code, it is 90 /as on the CHI 


computer, while a Fortran code requires 230 /ts on the 
IBM360/91. The number of floating point operations is 101 
multiplications, 92 additions and 1 division for the 2 V 2 code 
and 184 multiplications, 174 additions and 1 division for the 
3D code. Both codes reach an effective speed of about 4 
megaflops. 
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The storage requirement, processing timing and other sta¬ 
tistics of the 2 V 2 D code is given in Table III. This code is 
an isothermal one, in which temperature is a constant, so 
there are seven equations instead of eight. The AP main 
memory is big enough to store data for a system as large as 
80 2 . 

In case of the 3D code, disks must be used; for instance, 
it requires two megawords of data in a 64 3 system. In each 
grid point calculation, eight variables defined on the grid 
point and its six nearest neighbor points, or a total of 56 
numbers are used. However, some of these data can be 
saved for calculations at the next grid point in the column 
or adjacent grid point in the next column, so the need of 
data transfer is reduced. The technique of parallel processing 
between the IOP’s and the AP is used. 

Comparison with results on the IBM360/91 

The comparison of these plasma simulation codes on the 
CHI system and the corresponding Fortran (or Assembly) 
codes on the IBM360/91 are given in Table IV. For executing 
a one-million-particle simulation or a 64 3 3D MHD code on 
the IBM360/91, we assume disks are used for storage. The 
I/O charges are based on this assumption. 

The speed of the particle code is four times faster than 
the IBM assembly-language code. For the MHD codes, the 
speed advantage is not as good. Nevertheless, all codes 
attain an effective speed of 4 to 6 megaflops. Most impor¬ 
tantly, the CHI system shows a tremendous savings in total 
cost, with better than a two-order-of-magnitude decrease in 
cost from the IBM360/91. The figures for the IBM360/91 are 
actual costs if the programs were run at UCLA. For the 
CHI computer system, estimates are based on the assump¬ 
tion that computer time charge is directly proportional to 
the price of the computer system. 

In addition, we should point out that the CHI system 
achieves better accuracy than the IBM360/91 in single pre¬ 
cision. One example is that the kinetic energy calculated in 
the 2 V 2 D MHD code on the CHI machine is two digits more 
accurate than on the IBM360/91 in single precision. 

CONCLUSION 

We have presented a general description of the UCLA CHI 
computer system and have emphasized its special features, 


namely, two levels of parallelism in the system. Parallel 
processing among the AP, the MP, and the IOP’s allows 
execution of large-scale problems, and a high degree of 
internal parallelism within each processor allows fast oper¬ 
ations at very moderate cost. In addition the CHI system 
provides a time sharing operating system and supports a 
high level language, the Math System, which is particularly 
useful in solving physics related problems. 

The test results for three plasma simulation models clearly 
show the capabilities of the system. It performs high speed 
execution of large problems at relatively low cost. Since the 
plasma simulation models are similar in structure to many 
simulation models in other research fields, the CHI com¬ 
puter system is expected to have a wide range of applica¬ 
tions. 8 
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Multiple microcomputer systems for 
solving certain fluid flow problems 

by JOHN STEINHOFF 
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Bethpage, New York 


INTRODUCTION 

In the last several years, minicomputer based systems have 
been considered or used for an increasing number of sci¬ 
entific computing tasks. For example, Kottler and McGill 1 
used a minicomputer system to solve a series of problems, 
including a partial differential equation for pollutant disper¬ 
sal. Also, in 1973, Steven Orszag 2 compared a minicomputer 
system to large general purpose machines for solving some 
time dependent hydrodynamics codes. The conclusions 
were that minicomputers could be more cost effective than 
the large machines. In particular, Orszag concluded that the 
minicomputer was more cost effective than a CDC 7600 but 
less than a CDC STAR for moderate resolution hydrody¬ 
namics codes. For these problems, the data bases are large 
and the system is essentially a disk based one, where data 
flows from a large data base on the disk to the computer. 
When these studies were done, disk speed was not a restric¬ 
tion and the limiting factor was the computational speed of 
the minicomputer. 

In the last several years, improvements in LSI technology 
have made it possible to assemble reliable computational 
devices with a much lower cost per computation than mini¬ 
computers. These devices, ranging from moderately priced 
(—$100,000) fast systems with tightly coupled multiple func¬ 
tional units to very inexpensive simple units can be very 
cost effective when used with a minicomputer as host. They 
have had a large impact on the signal processing field, 3 in 
some cases proving to be faster than large machines such as 
the CDC 7600. They can also be effective for real-time 
simulation, as described in Reference 4. 

In this paper we will describe a study that was done on 
the effectiveness of a minicomputer based system with mov¬ 
ing head disk and peripheral processor for a class of fluid 
flow problems. 

It is easy to see that a moderately priced system can be 
assembled that would have a (disk) data capacity and con¬ 
tiguous block throughput and a (peripheral) computational 
speed of a large multi-million dollar machine. The extent to 
which this potential can be realized for our problem depends 
on, among other things; 

• Whether data can be transferred between disk and a 


relatively small random access memory at close to con¬ 
tiguous-block rate during the computation. 

• Whether the elements of the peripheral processor can 
be kept operating efficiently (in parallel). 

• Whether the system can be programmed efficiently. 

We will describe a disk data allocation scheme that avoids 
excessive disk accesses and achieves a high data throughput 
rate, and a scheme to use a set of peripheral processors 
efficiently, for the problem considered. We will also describe 
progress that we have made in assembling and operating a 
prototype system. At this time we have not obtained enough 
programming experience to comment on the ease of use of 
the system. The types of problems that we are looking at 
will be run for long periods of time, justifying the extensive 
use of assembly language programs. Once the possibility of 
effectively solving these problems has been established, it 
will be worthwhile to develop software routines which can 
be used in higher level language programs. 

Finally, we will describe a scheme to assemble a very fast 
system consisting of a set of modules arranged in a simple 
configuration. This system will have the capacity and speed 
to economically solve some very large problems. 

DYNAMICAL EQUATIONS 

The basic problem involves the solution of turbulent flow 
of a viscous, incompressible fluid, using the Large Eddy 
Simulation approach. In this approach, the Navier Stokes 
equations which describe the detailed dynamics are first 
spatially filtered resulting in a damping of the high spatial 
frequency modes of the turbulent flow. After removal of 
these modes, which cannot be resolved on feasible com¬ 
puters, only the resolvable “large eddies” are treated ex¬ 
plicitly. A sub-grid model is used to compute the stresses 
caused by the removed “small eddies.” The resulting equa¬ 
tions, when discretized, are tractable on current very large 
computers. The approach leads to good predictions for some 
quantities, such as the filtered energy spectrum in isotropic 
turbulent decay. 

Our initial approach will be similar to that of the Stanford 
Group. 5 * 6 Most of this work has been done on a CDC 7600 
and we should have a good basis for comparison. 
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The equations that we are solving are 


dlii dP 

dt Vi dXi 


(1) 

to 


(2) 

dx { 2 dXi Vi 


_ d 2 \ (dui 

Vi— (l-c 4 Y) u > (ta, 

_ du i\ 

dXi) 


d d 2 


(3) 

dxj Tii + V d Xj 2 Ui 



where u x are the filtered velocities, P is the pressure, v is 
the molecular viscosity, r y the model term for the high 
frequency modes and C A is a constant depending on the 
filtering function. The most popular model for r i} is based 
on the Smagorinsky model: 


Tjj — ~2 V T Sy 



V T =C(2 SySij) 112 . 


In the above equations, repeated indices imply summation. 

The basic approach is to use (1) and (3) to compute u t at 
a new time step, with P computed using (2). This method 
enforces incompressibility by predicting zero divergence for 
Ui at a new time step, if u { has zero divergence at the current 
time step. The Adams-Bashforth method, which is second- 
order accurate in At, is used to integrate u t : 


n dut in) 

Wi («+l)= Wi (n)+A t 


1 dUj in - ir 

2 dt 


(4) 


Various finite difference methods have been used to ap¬ 
proximate the spatial derivatives. Our example will involve 
second order equations for the filter related terms (contain¬ 
ing C A ) and fourth order terms for all other derivatives. 
Other difference formulae, involving a fixed number of 
neighboring points, can be computed on our system with 
schemes similar to the one we will describe. In one dimen¬ 
sion, the approximations to the first derivative are: 


8a(;c)= yy [ a (*+Ax)-a(*-Ajc)] 


Sa(x)= i2Ax t“ a ( JC+ ^Ajc)+8a(jc+AA:) 

- 8 a (x- Ax) + a (x - 2 Ax ) ] 

Our initial problems will involve homogeneous flows. The 
periodic boundary conditions used in these flows are simple 
to handle and represent a first study for our system. Other 
boundary conditions, such as no-slip ones, result in addi¬ 
tional problems from the numerical analysis 7 as well as pro¬ 
gramming point of view. Especially when higher order meth¬ 
ods are used, different operations must be done at grid 
points near boundaries. Unlike array processors, our system 
will be able to do a number of these different tasks in par¬ 
allel, but the programming will be more difficult. For the 


periodic boundary conditions used, the Poisson equation for 
the pressure can be solved directly using three-dimensional 
Fourier transforms. 


COMPUTING SYSTEM 

The basic computing system considered for the study con¬ 
sists of a moving head disk, a minicomputer and a set of 
bipolar microcomputers connected to the minicomputer by 
a time-multiplexed bus (see Figure 1). A prototype system 
is currently being constructed in the Research Department 
at Grumman. 

The capacities of available minicomputer disks run up to 
300 megabytes (Trident T-300) at a cost of about $20,000, 
with (contiguous block) transfer rates up to about 500,000 
bytes per second. The access times are about 10 to 70 mil¬ 
liseconds depending on how many tracks the head must pass 
over. The particular disk used in our system has a 10 meg¬ 
abyte capacity and 160,000 byte/second transfer rate. For 
the problem considered, the disk characteristics are very 
important. 

The host minicomputer used in the prototype system is a 
Data General NOVA 800, with a random access memory of 
32,000 16 bit words and an 800 ns. basic cycle time. It is not 
clear whether a higher performance minicomputer would 
result in higher throughput for our problem. Although the 
NOVA 800 cost more when purchased, minicomputers with 
similar performance now cost less than $10,000. 

The peripheral processor considered is of the type being 
constructed in the Research Department. It consists of a set 
of independent 16 bit machines, each with a cycle time of 
180 ns., separate high speed cache data memory (IK word) 
and microprogram memory (2K word) together with a larger 
16-32K MOS data memory. For our problem, no direct in¬ 
termodule communication is necessary and a single time- 
multiplexed data bus allows each microcomputer, when en¬ 
abled, to communicate to the host via direct memory access 
(DMA). The number of microprocessor modules that can be 



Figure 1—Computing system organization 
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used effectively depends mainly on the disk transfer rate for 
available disks. The cost of each module is about $4,000. 


IMPLEMENTATION 
Data flow 

The grid that we are using contains (64) 3 nodes (the largest 
size run on the 7600) and the resulting data base for the 
problem is at least several million words, depending on how 
it is decomposed. Hence, for a modest system such as ours 
a moving head disk must be used for data storage. For 
available disks, the access time is three orders of magnitude 
longer than the time required to transfer a (32 bit) word. 
Also, this transfer time is an order of magnitude longer than 
the read time for the random access memories used in the 
system. Thus it is essential to use a data allocation scheme 
that minimizes disk accesses, and an algorithm that mini¬ 
mizes the total number of variables transferred to and from 
disk each time step. 

For our problem, we can reduce the disk variables to two 
three-vector arrays and a scalar array: 

Ui n) , i>/ M) and P n \ 

where 

Mj (n) = Wj <n) +§ AtVi™ — + /* n-1) 

C/Xf 

and Vi (n) and i* B) are defined in (3) and (2). Then, 
u ( n +o=Ui (n) -%At^- F n) 

2 dXi 

In the above, /* n) is the one-dimensional Fourier transform 
of /* n) (the reason for storing P n) and not P n) will be ex¬ 
plained below), and n refers to the time step. 

The computations required at each time step can be di¬ 
vided into two classes: a semi-local class where finite dif¬ 
ference approximations to the partial derivatives are com¬ 
puted and the velocities are updated; and a global class 
where Poisson’s equation is solved by a direct method. The 
data flow requirements for these two classes are quite dif¬ 
ferent. Data allocation and computational schemes have 
been formulated that require a relatively small number of 
disk accesses and a near-minimum number of variables to 
be read from and written to disk for these classes of com¬ 
putations. 

First, the data allocation scheme will be described. The 
«i, Vi and P arrays are partitioned into “records” that span 
4 units (out of 64) in the x t direction, 8 units in the x 3 
direction and all 64 units in the x 2 direction (see Figure 2). 
These records are arranged on disk in 16 segments (8 records 
each) that correspond to different sets of x t values and all 
64 x 3 values, together with extra “scratchpad” space (for 
two records). Although the segments are arranged sequen¬ 
tially on the disk, they are depicted in Figure 2 next to each 
other. In the figure, the index represents the position of the 
record in the x x -x 3 plane. The positions are shown for a 


particular stage in the sweep through the Hi array, to be 
described below. 

There are three computational phases at each time step. 
In the first phase, ii i (n) , vfl l \ P n) are read from disk and « 4 (n+1) , 
tV n+1) written back. This phase requires 8 sweeps in the x t 
direction (for different x 3 values), each sweep taking 16 
steps (see Figure 3). At each step a set of /Y n) values are 
read from disk and i Xn) computed by Fourier transform in 
the x 2 direction. Then 8i/* n) is computed, «i (n) read and 
u { (n+1) computed. Then T a and v { (n+1) are computed. Finally, 
Vi (n) is read from disk and u { (n+1) computed and written back 
to disk together with y i (ra+1) . It is easy to see that, since each 
record contains all 64 values of x 2 , once these are in random 
access memory any partial derivatives as well as Fourier 
transforms can be computed in this direction. Also, x t de¬ 
rivatives can be computed efficiently by storing some values 
from the previous steps since the sweep is in the x x direc¬ 
tion. Only for the x 3 derivatives do we need to read some 
variables more than once: As pictured in Figure 3 for each 
sweep, a band must be read that is wider than the width of 
the sweep (8x 3 units). Also, we cannot write over variables 
during a sweep if the old ones are needed again for the next 
sweep. This is illustrated in Figure 2, where the positions of 
the updated « 4 <n+1) records and the old « 4 <n) records are 
shown with our scheme. Only one disk access is needed to 
read the «/ n) variables required for the next step (including 
overlays) and write the updated ones. Also, at each step one 
access is required to read the P (n) variables (including 
overlap) and one to read and write the u/ ra) variables. (Steps 
near the edges of the grid require extra accesses because of 
the periodic boundary conditions). 

The second phase required at each time step involves 
computing p (n) =S i u i (n) , the source function for Poisson’s 
equation (2). The procedure used in the first phase is used 



0 


'-l-1-i-1-|-1 3 

i 0,0 ' o,i ! 0,2 ; 0,3 j 0,4 0,5 '-j 

j__1 1_I 1_1 < 



oft 

£l 

0,2 

oft 

—— 

0,4 

0,5 

0,0 

1.0 

Cl 

L2 

CS 

C4 

1.5 

1,0 

2,0 

£l 

1,3 

1,4 

1,5 

1,6 

2,0 

2,1 

2.2 

2,3 

2,4 

2,5 

2,6 

3.0 

3,1 

3,2 

3,3 

3,4 

3,5 

3,6 

4,0 

4,1 

4,2 

4,3 

4,4 

4,5 

4,6 

5,0 

5,1 

5,2 

5,3 

5,4 

5,5 

5,6 

6,0 

6,1 

6,2 

6,3 

6,4 

6,5 

6,6 

7.0 

7,1 

7,2 

7,3 

7,4 

7,5 

7,6 

0,0 

0,1 

0,2 

0,3 

0,4 

0,5 

0,6 


0 1 2 3 4 5 6 

SEGMENT 


DATA ALLOCATION: 
Figure 2 






412 


National Computer Conference, 1978 


ARRAY LOCATION OF DATA READ FROM DISK AT ONE 
COMPUTATIONAL STEP 



Figure 3—Array location of data read from disk at one computational step 

here, but the p (n> data is first Fourier transformed in the x 2 
direction before being written onto disk (as p <n) ). 

The third phase involves taking x x and x 3 transforms of 
the p (n) array, dividing each element by a (constant) function 
of position, and taking inverse x t and x 3 transforms. The 
Poisson solution is complete when the transformed variables 
(now denoted /* n) ) are transformed in the x 2 direction at 
each step as they are read (in the first phase of the next time 
step). In the third phase, essentially, a transpose must be 
done so that entire *i-x 3 planes of data can be brought into 
random access memory. 

There is an extra degree of freedom in our disk addressing 
that was not used in the first two phases, but which will be 
important here: The disk has four separately addressable 
surfaces. Thus, if data corresponding to all x 2 values is 
properly arranged on the four surfaces, then we can read a 
set of entire Xi-x 3 planes corresponding to 1 U of the differ¬ 
ent x 2 values by reading one surface. This can be done at 
close to contiguous-block rates: where all the variables 
that pass under the one recording head are used. The total 
reading (or writing) time is essentially the same whether we 
are reading V 4 of the p (n) array, or any smaller fraction. For 
a system with 96K 16 bit words of random access memory, 
we can store 3 /ie of the array and essentially read and write 
the variables at 3 A contiguous-block rate. The total number 
of variables of each type transferred to and from disk to¬ 
gether with the total number of accesses and timing esti¬ 
mates are given for the three phases in Table I. 

Microcomputer operation 

The microcomputer modules each have an independent 
instruction, data and scratchpad memory. Most of the ran¬ 


TABLE I.—Disk Transfers 


PHASE 

VARI 

ABLE 

AC 

CES 

SES 

WORDS 

READ 

WORDS 

WRITTEN 

TIME 

1 


152 

2.9x10 6 

1.8x10 6 

62 sec 
(26) 


T 

160 

1.3 

0 

23 

(13) 


Vj 

128 

1.5 

1.5 

42 

(18) 


TOTAL 

340 

5.7 

3.3 

127 

(57) 

2 

V. 

1 

72 

2.1 

0 

28 

(12) 


'■N*' 

P 

32 

0 

.5 

8 

(4) 


TOTAL 

104 

2.1 

.5 

36 

(16) 

3 

y 

6 

.8 

0 

9 

(3) 


P 

6 

0 

.8 

9 

(3) 


TOTAL 

12 

.8 

.8 

18 

(6) 


*( ) REFERS TO FAST DISK 


dom access memory resides in the microcomputer data 
memories. A simple control scheme that we are currently 
implementing on the prototype system (with two modules) 
involves a time-multiplexed bus attached to each microcom¬ 
puter. The host (minicomputer) selects each module in turn 
by enabling its output buffers. The selected module transfers 
data to and from the host by DMA. When the transfer is 
complete, based on a signal from the module, the host se¬ 
lects the next one and the module starts computing. 

The arithmetic operation count for the three phases is 
given in Table II in terms of the number of (one-dimensional) 
FFT’s, square roots, divisions and other additions and mul¬ 
tiplications. The total computing time estimates for one 
module are also given assuming characteristics similar to the 


TABLE II.—Computations 


P 

H 

A 

S 

E 

FFT 

SORT. 

DIV. 

MULT. 

ADD. 

TOTAL* 

1 

4 X 10 3 
(REAL 
64 PT.) 

.25 X 10 6 

0 

12 X 10 6 

50 X 10 6 

245 SEC. 

2 

4 X 10 3 
(REAL 
64 PT.) 

0 

0 

.25 X 10 6 

6 X 10 6 

32 SEC. 

3 

8 X 10 3 
(COM¬ 
PLEX 

64 PT.) 

0 

.25 X 10 6 

0 

0 

90 SEC. 


*TIME FOR ONE 16 BIT MODULE 
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ones used in our system. It can be seen that the throughput 
of —2 modules will approximately match the disk transfer 
rate for our prototype system, and —26 modules for a system 
with a faster disk. 

A simple module data allocation scheme, to be used in 
our system, involves partitioning the data set at each step 
so that each module has data corresponding to the same x 1 , 
x 3 values but different sets of x 2 values (with the exception 
of the Fourier transforms in the x 2 direction, which will be 
done separately in each module.) As in the disk allocation 
scheme, some «i (n) and /* n) data must be accessed twice, 
but from the host instead of the disk and by two separate 
modules. This redundant data set is directly proportional to 
the number of modules used, and is approximately 7 K (16 
bit) words per extra module. A 4 K buffer space is also 
necessary for each module. Thus, the total memory require¬ 
ments increase by —11K for each module added. For 
systems with more than several modules, an alternative 
to redundant storage would be to use inter-module 
communication. The time-shared bus will become saturated 
if used with more than —6 modules. Beyond this number, 
a ring network with direct intermodule transfers would be 
a very efficient scheme. 

Assuming buffered operation, the total time for one time 
step will be about 210 seconds for our system with 2 modules 
and 90 seconds for a system with a faster available disk and 
6 modules. The time quoted for a similar algorithm (coded 
in Fortran) on a CDC 7600 is 180 seconds per time step. An 
assembly language coded version should run several times 
faster. Between 50 and 200 time steps are required to resolve 
a significant portion of turbulent kinetic energy decay in 
many of these simulations, resulting in a total solution time 
of several hours per case. 

A straightforward way to double the performance of the 
system would be to use two disks and either 12 16 bit 
modules or 6 expanded 32 bit ones. Such a system, with 300 
megabyte disks would be able to solve (256) 3 problems, with 
200 time steps taking —5 days; a reasonable time for a 
moderately priced, dedicated system. 

MULTI-MODULE SYSTEMS 

We looked at ways to increase the solution speed for our 
problem by utilizing more modules, and at ways to increase 
the grid size (data base) by utilizing more disks. It turned 
out that a very simple configuration consisting of 8 of the 
systems described above could be used to give a speed 
increase of —7 and a data base increase of 8. Our reason for 
looking at larger systems was mainly to give an example of 
a highly parallel implementation of a particular flow prob¬ 
lem. If other types of flow problems prove to be amenable 
to this type of architecture, or if the system cost decreases 
enough in the future, it may be worthwhile to build such a 
system. 

We define a unit to be a minicomputer, set of microcom¬ 
puters (16 or 32 bits) and one or two disks as described 
above. The configuration studied is quite simple: we just 
have to dual-port each disk and form a ring configuration 
(see Figure 4). For the (64) 3 problem, 8 sweeps (in the x t 


direction) were required for a single unit for the first two 
phases of each time step. We could then use 8 units—one 
for each sweep. With some minor exceptions, each module 
could read the same variables (during a sweep) as if it were 
doing the entire problem. These variables would be available 
on the two disks (or four disks) connected to each minicom¬ 
puter. If the disk transfers were coordinated so that each 
left-hand disk were accessed at the same time by each min¬ 
icomputer and then each right-hand disk, there would be no 
disk conflicts. This synchronization would require the test¬ 
ing of a flag only after the transfer of several hundred vari¬ 
ables, and could be done using straightforward software 
techniques and standard interprocessor busses (IPB’s). An 
advantage of this scheme is that one unit by itself could use 
essentially the same program for these phases and solve the 
entire problem. Also, by changing the (x 3 ) width of the 
sweep of each unit and the number of sweeps each unit must 
take, we could conveniently use any number of units up to 
8 . This should simplify development of the full system and 
decrease time lost due to failures. The computational speed 
should increase roughly linearly (in these phases) with the 
number of units. 

We now describe a scheme for the third phase. For the 
single unit, approximately 10 percent of the disk transfer 
time was associated with this phase. With 8 units, we can 
segment the data (p array) in the x 2 direction into 8 groups 
and rearrange the groups among the units. Each unit then 
would have to transfer 7 groups to the other 7 units. By 
transferring around the ring first in one direction then in the 
other, the rearrangement can be done in seven steps. During 
the operation, each piece of data moves an average of 2 
units and we finally have 8 entire (x!-x 3 ) planes of data in 
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Figure 4—Multi-module system 
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Figure 5—Transpose—First step 

each unit (corresponding to different x 2 values). The flow of 
the data (in one direction) is depicted in Figure 5. Each solid 
line represents a transfer at the first step. The number of 
groups transferred by each unit is 4, 3, 2 and 1 for the 
respective right-steps, and 3, 2 and 1 for the left-steps. (Most 
of these transfers can be overlapped with the first two 
phases.) The entire phase should take about 3 /s as long as 
for a single unit. To complete the phase, we then have to do 
a two-dimensional (r,-r 3 ) transform in each unit, a division, 
an inverse transform, and then an inter-unit transfer again. 
The time required for the total time step (all three phases) 
is ~6 sec., compared to 44 sec. for one (dual disk) unit, 
giving a speedup of ~7. 

For a (256) 3 array, we have 640 and 320 seconds per time 
step, respectively, for single and dual disk systems. For 200 
time steps, this gives —40 and 20 hours. Dual 300 Mbyte 
disks and 6 32 bit microcomputers would allow a (512) 3 
solution (200-400 time steps) in 6-12 days. 

There are some advantages to using the system that we 
have described. First, since it is a one-dimensional config¬ 
uration with simple connections, adding new units or dis¬ 
connecting failed ones should be straightforward, both in 
software and hardware. This also applies to each unit, which 
is a linear configuration of microcomputers. Second, al¬ 
though we have 48-96 (32 or 16 bit) microcomputers operat¬ 
ing in parallel, only groups of 6-12 microcomputers need be 
coordinated, and only a group of 8 units need be coordi¬ 
nated. Third, each unit can have its own operating system 
and do an entire problem by itself, with any number of 
microcomputers. 

This type of simple arrangement is very efficient for all 
the operations required except the one-dimensional Fourier 
transforms in the x s direction. Since a relatively small 
amount of data is associated with this operation, the overall 
system is efficient. 


DISCUSSION 

We have described a series of machines, assembled from 
available subsystems, that can have a throughput ranging 
from a fraction of that of the CDC 7600 to an order of 
magnitude greater than the 7600 for a specific time depend¬ 
ent three dimensional flow problem. The cost of the ma¬ 
chines range from approximately $50,000 to over $500,000, 
and, for each, the cost for a solution to the problem studied 
is almost two orders of magnitude less than for the 7600. In 
these machines, extensive use of buffering and software 
synchronization can be made so that inter-component con¬ 
nections are very simple. Also, the systems can be highly 
modular, with more powerful machines consisting of sets of 
simpler machines connected together. Thus, a program to 
develop machines along these lines and gradually increase 
complexity and computational speed should be fairly 
straightforward. 

The systems use sets of microcomputers to do most of the 
computations, and the problem is decomposed into parts to 
be solved independently by each microcomputer, as if it 
were doing the problem sequentially. This “macroparallel” 
approach should be compared to other approaches, such as 
one where sets of tightly coupled functional units operate 
concurrently on a single data stream. The AP8-120B periph¬ 
eral processor is the most popular version of this. 

We have described specialized algorithms to solve our 
problem on the machines considered. The main part of these 
algorithms concerns data flow. We found that special atten¬ 
tion must be paid to this if effective use is to be made of the 
moving head disks. Use of these disks will be necessary, at 
least over the next few years, for economical storage of very 
large data bases in modest systems. 

The problem studied represents a specific type of com¬ 
putational method (two time level, explicit) and a particular 
set of boundary conditions (periodic). Other methods (e.g., 
implicit) and boundary conditions should also be considered. 
Like our method, versions of implicit methods have semi¬ 
local and global classes of computations. Also, these global 
classes often involve solutions of one-dimensional tridi¬ 
agonal equations in three independent directions, and a com¬ 
parison with our study would be interesting. 

Finally, software will, of course, be very important if 
wider classes of problems are to be solved. Since the mi¬ 
crocomputers operate sequentially, higher level language 
implementation should not be too difficult, although efficient 
use of the small high speed control stores may prove other¬ 
wise. Another difficulty appears to be in the efficient use of 
the disk. Perhaps classes of data flow algorithms can be 
coded and used with a higher level language. 
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Applications of minicomputers to Library 
Systems and Services 


SESSION CHAIRMAN—LAURA J. RAINEY 
Rockwell International 

Panel Members 

Audrey N. Grosch—University of Minnesota 
Charles M. Goldstein—National Library of Medicine 
Karl M. Pearson, Jr.—California Library Authority for 
Systems and Services 


SELECTING A MINICOMPUTER SYSTEM—Audrey N. 
Grosch 

Factors influencing the choice of minicomputer systems 
for library automation will be discussed. Specific conditions 
to be treated will be: 

Whether already tailored, packaged systems provide the 
necessary functional capability; 

Whether program development and hardware systems will 
be configured and produced by one system vendor or by 
custom programming by a firm specializing in program¬ 
ming for a given manufacturer’s smaller or larger config¬ 
urations; 

Whether the library has in-house capability for hardware/ 
software evaluation as opposed to using contractors or 
consultants for this choice; 

Whether total program development for applications will 
be done using commercially available DBMS (Data Base 
Management System) software packages such as IMAGE, 
MUMPS, DBMS-11; 

Whether parent organization has standardized toward one 
manufacturer for machines to be used in connected mode 
to a large main frame computer; 

Whether the system will be a stand alone or remote link 
to a main frame computer or a shared use of a larger 
minicomputer configuration having a multi-user, multi¬ 
tasking operating system. 

Some characteristics of minicomputers presently avail¬ 
able, together with a description of the range of peripheral 
devices will also be presented along with some procurement 
guidelines. 


LIBRARY AUTOMATION PROGRAM AT THE 
LISTER HILL NATIONAL CENTER FOR 
BIOMEDICAL COMMUNICATION— 

Charles M. Goldstein 

Lister Hill National Center for Biomedical Communica¬ 
tion at the National Library of Medicine is supporting an 
R & D program in library automation. One aim of this 
program is to develop an integrated minicomputer based 
library system to satisfy the following objectives: 

Transportability 

Expandability 

Multiple levels of on-line user 
interbase and systems network access 

Transportability refers to the ability to implement the sys¬ 
tem on more than one type of computer as well as easily 
maintain the program, to wit, a higher order language im¬ 
plementation. Expandability is defined as the ability to re¬ 
spond to increased loads by scaling up the equipment with¬ 
out changes in the program. Systems network access implies 
automatic access to network resources by the system with¬ 
out manual intervention on the part of the user. An example 
of the latter would be automatic access to a network avail¬ 
able catalog file, if an item is not found in a local file. 

The initial modules under development are circulation and 
acquisition modules. The potential impact of such systems 
on small, medium, and large libraries, as well as their po¬ 
tential impact on a National Library Network architecture 
will be discussed. 


MINICOMPUTERS IN LIBRARY NETWORKING— 

Karl M. Pearson, Jr. 

Shortly after minicomputers appeared in library applica¬ 
tions as the base for turnkey circulation control systems, 
their value for networking applications was realized. First, 
they have been put to use as front-end processors and com¬ 
munication controllers for the centralized, on-line catalog 
systems, most notable, OCLC and BALLOTS. Next, they 


417 



418 


National Computer Conference, 1978 


have begun appearing in regional service centers. The New 
England Library Network (NELINET) has installed a mini¬ 
computer as an interface between member libraries and 
OCLC. In addition to serving as a data concentrator and 
communications handler, experiments have been run using 
the minicomputer as a sophisticated message store and for¬ 
warding device to facilitate interlibrary loan. 

Several libraries have procured minicomputers for general 
in-house data processing needs, particularly for economical 
text editing, data checking, and on-line file maintenance. 


Many of these computers also serve in a distributed proc¬ 
essing role in which they can link to a host maxicomputer 
for large-scale processing requiring extended sorting and 
formating of large bibliographic records. 

Future developments are likely to see a minicomputer 
based communication network to pass bibliographic data 
from one utility to another, (e.g., between OCLC and the 
Library of Congress), to transmit interlibrary loan transac¬ 
tions, and allow user access to any library data base regard¬ 
less of the type of terminal being used. 



PART II—METHODOLOGY 




Area Director: 

Stephen R. Kimbleton 
National Bureau of Standards 
Washington, DC 


Performance measurement and evaluation 


Computer systems performance analysis has matured rapidly within the last 
decade. Ten years ago, the field was just beginning to gain prominence. Results 
were sparse and general methodologies were lacking. The efforts of individual 
researchers were exciting but difficult to interrelate. 

The present state-of-the-art performance is a dramatic improvement over that 
of ten years ago. Although the performance problem cannot be regarded as 
solved, a very significant degree of progress has been made. General purpose 
tools are transitioning from the feasibility investigation stage to the prototype 
stage. Major components of a possible structure for the field can be discerned 
and are reflected in the following discussion. 

The four performance analysis sessions at the 1978 National Computer Con¬ 
ference were structured to provide a unique opportunity to learn about some of 
the most exciting current developments in the field. The presentations and their 
associated papers describe some surprisingly sophisticated tools which are cur¬ 
rently at the prototype stage together with a context for their appreciation. 

NCC78 performance sessions covered a spectrum of issues from enduser and 
installation manager requirements to theoretical developments which are the 
prerequisite to future implementation of even more sophisticated tools. To ac¬ 
complish this, the sessions were structured to focus on: Computer Performance 
Management, Computer Performance Technology, Computer Performance Mod¬ 
eling, Mathematical Analysis of Computer Performance and End User Perform¬ 
ance Considerations. 

COMPUTER PERFORMANCE MANAGEMENT 

Evaluating performance from the viewpoint of an enduser or installation man¬ 
ager is complicated by two factors. The first is the immense volume of perform¬ 
ance statistics which can be collected using current hardware or software moni¬ 
tors. The second is the problem of handling the collected information. The 
Computer Performance' Management session, chaired by Phil Kiviat, focused on 
these two problems. 

Solving the first problem requires prior determination of those performance 
measures which are relevant for the needs of endusers and installation managers. 
In contrast with other areas of science and technology, the need is not for more 
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data but for the right data. Presentations and the associated papers concerned 
with this issue covered both the types of measures which are useful and their 
relationship to the dependence between data processing staff and data processing 
users. 

Given that the desired data has been identified, the next requirement is auto¬ 
mating its handling. This requires a management information system for perform¬ 
ance data. Major issues which must be considered in the development of Per¬ 
formance Management Systems are described in the context of recent work in 
this area. 


COMPUTER PERFORMANCE TECHNOLOGY 

Unacceptable system performance requires changing either the system or its 
workload. Provided that the system and its workload can be appropriately char¬ 
acterized, performance prediction techniques can be used to evaluate the likely 
results of specific changes and to answer “what if’ questions. 

The Computer Performance Technology session, chaired by Steve Kimbleton, 
described new techniques for characterizing the workload of a computer system 
through classifying it into clusters. In addition, a new approach for characterizing 
individual jobs was described. Taken together, these two characterizations pro¬ 
vide a more effective means for developing the required input information for 
performance prediction tools. A fast, user oriented queuing network based per¬ 
formance prediction technique was also described. 


COMPUTER PERFORMANCE MODELING 

Modeling plays a key role in evaluating “what if’ types of questions. Model 
development can be pursued via either analytic or simulation based approaches. 
Selection of either approach usually forces a tradeoff between the design insight 
provided by analytic models and the level of detail which can be reflected in 
simulation models. As a result, careful consideration of the goals of the model 
user is required. 

The Computer Performance Modeling session, chaired by Jeff Buzen, described 
the requirements of an installation manager for effectively using modeling tech¬ 
niques. A new hybrid approach based on combining simulation and analytic 
techniques thereby permitting both increased levels of detail and faster speed 
was also described. In addition, a model for investigating peripheral processor 
performance was also presented. 


MATHEMATICAL ANALYSIS OF COMPUTER PERFORMANCE 

Computer system models are a key source of design insight. Developing ana¬ 
lytic forms of such models requires close interaction between researchers in the 
areas of queuing and other types of stochastic processes and model oriented 
performance analysts. 

The Mathematical Analysis of Computer Performance session, chaired by 
Hisashi Kobayashi, described some recent results in the interdisciplinary area of 
systems modeling. New techniques for evaluating the performance of queuing 
systems were presented. These results included a new entropy based approach 
to obtaining the time dependent behavior of certain queues and the distribution 
of the cycle time of a closed queuing network. In addition, new results in 
evaluating the performance of processor scheduling algorithms were also de¬ 
scribed. 
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END USER PERFORMANCE CONSIDERATIONS 

System performance, as viewed by an end user, can be improved by enhancing 
the performance associated with a given program or, alternatively, by simplifying 
the user’s interaction with the system. This session chaired by Shirley Watkins, 
is concerned with aiding the end user in efficiently using computer resources 
through simplifying their use or evaluation. The first paper will describe a tech¬ 
nique for faster and simpler program design verification through using a data 
generator for program testing. 

The second paper looks at the end user-system relationship in the context of 
providing an adaptive system interface. These formal presentations will be com¬ 
plemented by an informal panel discussion. 




How to improve your performance 
through obfuscatory measurement* 

by DAVID F. STEVENS 

University of California 
Berkeley, California 


INTRODUCTION 

It is best, I believe, to make it clear at the outset that this 
monograph has a somewhat obfuscatory title ... for it is 
not my intent actually to show you how to improve your 
performance (that being a merely technical exercise devoid 
of true intellectual challenge), but rather to show you how 
to claim—and substantiate—superlative performance, even 
in the face of extreme user discontent. 

As I have noted elsewhere, 1 there are two basic elements 
of success in computer center management: the achievement 
of saturation and the demonstration of efficiency. That ear¬ 
lier paper was somewhat deficient in that it concentrated 
upon the one (achievement of saturation, independently of 
the true—users’—workload) and slighted the other; it is 
hoped that this effort will somewhat redress the balance. 

At this point I should, in all modesty, note that none of 
the techniques discussed here are my own creation. My 
small contribution consists only in the recognition of these 
scattered efforts as elements of a significant whole. I should 
be quite surprised, in fact, if all who read this treatise have 
not already either employed or encountered every measure 
discussed herein. What has heretofore been lacking is a true 
appreciation of the difference between a system viewed 
through the rose-colored glasses provided by these meas¬ 
ures, and that same system viewed in the cold light of reality. 
It is my belief that the craftsman who understands his tools 
uses them most effectively; it is my hope that, after having 
read this paper, you will have a more complete understand¬ 
ing of obfuscatory measurement: what it is, what it can do 
for you, and, perhaps, how to devise your own obfuscatory 
measures. To that end, then, I will define what I mean by 
“obfuscatory measurement”, provide some general rules 
and principles for the creation of obfuscatory measures, and 
give a number of examples to demonstrate the power of 
obfuscation and hint at the astonishing variety of forms it 
may take. 


* This work was done with support from the United States Department of 
Energy. The submitted manuscript has been authored by a contractor of the 
U.S. Government under contract No. W-7405-ENG-48. Accordingly, the 
U.S. Government retains a nonexclusive, royalty-free license to publish or 
reproduce the published form of this contribution, or allow others to do so, 
for U.S. Government purposes. 


OBFUSCATORY MEASUREMENT DEFINED 

“Obfuscatory measurement” is measurement which ob¬ 
scures that which it should illuminate. 

To be quite precise, any measurement which gives a false 
impression is obfuscatory; we are here concerned, however, 
only with those which allow DP Management to paint a 
brighter picture than that seen by the users, for it is only 
those which contribute to their—the DP Managers’—suc¬ 
cess. Although you may be unfamiliar with the term, you 
will discover that you are familiar with many of the meas¬ 
ures ... for many (if not, indeed, most) of the performance 
measures in common use today are obfuscatory. Succinctly 
stated, obfuscatory measures 

• measure the wrong things 

• measure the right things . . . wrongly 

• measure something else (i.e. other than that which they 
purport to measure) 

• measure nothing at all (or at least no meaningful thing) 

Furthermore, since systems, like people, respond to meas¬ 
ures by which they are evaluated, these measures contribute 
to success (as defined in Reference 1), by rewarding pessi¬ 
mal performance, thus increasing saturation. 

GENERAL RULES, PRINCIPLES, AND 

TECHNIQUES 

1. Select your measures with care—Not all measures are 
appropriate to all situations. You should neither attack 
the fly with the cannon, nor the elephant with the! 
feather-duster. Tailor your measures to the tractability 
of your users and the gullibility of your upper manage¬ 
ment . . . and always have a couple in reserve, just in 
case. . . . 

2. When in doubt, seek the advice of your mainframe 
vendor—Remember, your vendor cannot sell you ad¬ 
ditional equipment until your upper management is 
convinced of the saturation and effective utilization of 
your existing configuration. Furthermore, your vendor 
has a wealth of experience in dealing with upper man¬ 
agements just like yours. . . . Obfuscation is the very 
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essence of the salesperson’s art; as you seek legal 
advice from a lawyer, you should seek obfuscatory 
advice from your vendor. 

3. The easier a measure is to obtain, the more likely it is 
to be obfuscatory—This is one of the few cases known 
to modern science where Murphy’s Law operates in 
favor of the practitioner. (There is no great mystery 
here, however: it is often the generosity of the vendor 
which made the measure easy to obtain.) 

Two specific kinds of easy measure are worthy of 
individual mention: means and percentages. As we 
shall see in the discussion of Availability, suitable def¬ 
inition of the base can turn any measurement into a 
praiseworthy percentage. As for the mean, while it 
frequently lacks meaning, it often exhibits meanness. 
Even though a number of articles in the recent litera¬ 
ture (notably a pair by Gary Carlson 2 and one by Jain 3 
have emphasized the obfuscatory nature of “indiscrim¬ 
inate” use of statistical concepts, the mean is so be¬ 
loved by the average person that its utility is expected 
to continue relatively undiminished. One can still, for 
instance, report a favorable mean in preference to a 
realistic median in most circumstances. It is also often 
fruitful to ignore the distribution. More details on this 
technique may be found in Reference 4. 

4. Overextend analogies—Concepts which are meaningful 
in other fields can sometimes be transferred into the 
computer performance arena, where they are invalid, 
without loss of prestige. It helps, of course, if the 
concept is so familiar that it is accepted without ques¬ 
tion in its new context. MTBI is such a measure. 

5. Creative Definition—This is an indispensable element 
of the obfuscator’s arsenal ... for the most misleading 
percentage you can devise won’t help you unless you 
can convince someone that it measures something. If 
yours is an elementary situation, actual definition is 
not important: a catchy name is all that is 
required . . . (Remember “CPU efficiency”? was 
there anything efficient about it? A modem example is 
“depth of multiprogramming”.) 

If you find yourself in deeper waters, some measure 
of definition must be supplied . . . but it is best if it is 
either ambiguous or incompletely specified. (“Availa¬ 
bility” as “percentage of time available” is, as we shall 
see, an excellent example of this technique.) 

OBFUSCATION IN PRACTICE—SOME SPECIFIC 

EXAMPLES AT WORK 

Some of these measures, most particularly Availability 
and MTBI, have been discussed extensively elsewhere; 1 * 4 
inasmuch as they are among the most widely employed 
obfuscatory measures, however, they are included here— 
briefly—for completeness. We will look at four general 
classes of measures, giving at least two examples of each. 
This should provide an ample foundation upon which you 
can construct the obfuscatory program most suited to your 
specific situation. (The contents of this section are summa¬ 
rized in Exhibit I for quick reference, together with a listing 


of comparable honest measures—which are offered in the 
belief that it is as desirable to know your enemy as to know 
your friends.) 

1. Measures of Availability and Reliability 

(a) Availability 

Usually expressed as a percentage, “availability” 
is taken by the uninitiated to indicate the amount 
of time the system is usable, whereas in fact it 
indicates the amount of scheduled time the system 
is available to the computer center. By reducing 
the base to scheduled time a significant increase in 
percentage is obtained. It is further increased by 
including many periods of time when it is not, in 
fact, fully usable: start-up times, time spent re-run¬ 
ning lost or interrupted jobs, and time devoted to 
the “run-down” before a scheduled interruption. 
Exhibit II illustrates the cumulative effect of all 
these adjustments. It shows a week in the life of a 
one-shift operation, with one period of preventive 
maintenance (PM), a daily system development 
shot (SD), two unscheduled periods of down-time 
(15 minutes on Tuesday and an hour on Friday); 
start-up requires half an hour, and “run-down” 
starts a half-hour before system development time 
and an hour before the end of the shift. Naive and 
obfuscatory measures stand in rather sharp con¬ 
trast. 

(b) MTBI (Mean Time Between Interruptions) 

MTBF (the mean time between failures) is so well 
accepted as a reliability measure in engineering 
contexts that practically no one questions its DP 
analog, MTBI. That the causes of failure in the two 
fields are largely unrelated is largely ignored: fail¬ 
ures in mechanical systems are caused by wear and 
fatigue (to which software is impervious); failures 
in computing systems are caused by unexpected 
input (to which mechanical systems are rarely ex¬ 
posed) and trivial overflows (which, if they cause 
damage at all, cause trivial damage: will an over¬ 
flow on the meter crash a taxi?). The user-oriented 
measure which most closely corresponds to MTBI 
is the mean (or median) service interval. To see 
how they compare, we return to the sample week 
of Exhibit II. The mean service interval, even giv¬ 
ing full credit for the run-down periods, is 2.23 
hours (26.75/12), and the median is 2.5. The con¬ 
servative way to calculate MTBI is to divide 
“hours available” by “number of interruptions 
plus 1”: 32.75/3=10.9 hours . . . more than three 
times as long as the longest service interval. 

2. Measures of Throughput 

(a) Utilization 

When the obfuscator is asked for measures of 
throughput she has ample industry precedent for 
responding with measures of utilization. Utilization 
measures are advantageous because they reward 
ineffective programming (which is much easier to 
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Class 

Honest Measures 

Sample Obfuscatory Measures 
(and what they actually measure) 

Availability 

and 

Reliability 

True availability to the 
users. 

Mean and median service 
interval. 

Availability (% uptime) 

MTBI (mean scheduled time to 
crash) 

Throughput 

Work delivered (in user 
units). 

Utilization (resource occupancy) 

Efficiency (utilization) 

Turnaround 

Processing time vs. 
system wait time, 
by category. 

Interactive response time 

Turnaround time 

(as long as you stick to 
means, they measure nothing 
meaningful) 

General 

Productivity 

Concurrency: the number 
of CP and channel 
activities occurring 
simultaneously. 

Existence (or not) of 
saturation. 

Capacity relative to 
workload. 

Functions delivered; 
quality of code. 

Overlap (existence of overlap) 

Depth of multiprogramming 

(number of active initiators) 

Saturation (work, productive or 
not, as % of "capacity") 

Lines of code (prolixity) 

Exhibit I 


obtain than the other kind). Your path here is not (b) Efficiency 

quite as free as it used to be, what with the intro- This is actually a vestige of the past, but one which 

duction of distinguishable “system” and “prob- has validity in some contexts, and adds a certain 

lem” states for CPU utilization . . . but it remains panache to your reports. It may be freely used in 

the case (thanks to your friendly mainframe ven- place of “utilization” (the two are identical in 

dor) that much of what is called “problem state” meaning), but I would advise against using them 

is actually system overhead. And it seems ex- both to refer to the same quantity: such a juxta- 

tremely unlikely that anyone is going to come up position might inspire tiresome questions. “CPU 

with a meter which distinguishes between “sys- utilization” and “channel efficiency”, on the other 

tern” and “problem” channel activity states! hand, provide a nice appearance of breadth. 
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Exhibit II 


SH 

PEEEM 

6/6 

35/8 

5.75/7 

3.75/8 

7/7 

4.5/8 

7/7 

4.5/8 

6/7 

3/8 

32.75/34 

19.25/40 

96% 

48% 


3. Measures of Turnaround 

(a) (Interactive) Response Time 

This has superseded “turnaround time” as the 
most commonly quoted measure of turnaround, but 
the principles of use are the same. It is most im¬ 
portant here to remember that the mean can be 
manipulated by stacking the extremes. To be spe¬ 
cific, you can achieve essentially any response time 
you wish by requiring a suitable number of trivial 
interchanges—with zero response time—to take 
place during any interactive session. (This is easily 
seen to be the inverse of the situation shown in 
Exhibit III, wherein a Miraculous improvement in 
“mean service interval” is achieved by introducing 
an unused—and hence uninterrupted—weekend 
service.) 

Another fact to be borne in mind is that, in some 
situations, response which is too quick creates ten¬ 
sion, which causes errors . . . and errors lead to 
wasted work, thus bringing saturation (and hence 
success) ever closer. (A better strategy, however, 
is to strive for consistently unexpected response 
time, whether it be quicker or slower than antici¬ 


pated . . . but this is somewhat off the subject of 
this paper.) 

(b) Turnaround Time 

Since the good turnaround times are the small ones, 
this is a situation where the median, surprisingly 
enough, favors the obfuscator. Nevertheless, I rec¬ 
ommend sticking with the mean. For not only is 
the median a dangerous precedent to set, the mean 
is, as we have seen above, quite a tractable index. 
As in the case of response time the enterprising 
manager can cause enough small jobs to be sub¬ 
mitted to achieve whatever mean turnaround time 
is deemed necessary. 

4. Measures of General Productivity 

(a) Overlap; Depth of Multiprogramming 

These two measures are grouped together not be¬ 
cause they are thought to be equivalent (they are 
not), but because they address the same problem: 
a vague understanding on the part of upper man¬ 
agement that some multiplicity of processing is de¬ 
sirable. They make a good combination, not only 
because they obfuscate in different ways, but also 
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because the two together give no more accurate a 
picture than either one singly. 

Overlap is in fact somewhat less obfuscatory 
than depth of multiprogramming, for it measures 
the percentage of time that some amount of simul¬ 
taneity is experienced; it does not, however, con¬ 
sider the level of simultaneity. (Thus two simulta¬ 
neous processes are every bit as good as seven.) It 
may be this very touch of honesty, paradoxically, 
which makes overlap so useful as an obfuscatory 
measure. 

Depth of multiprogramming, on the other hand, 
is pure obfuscation: it counts initiators instead of 
processes. In many shops, large values of depth of 
multiprogramming survive as tribute to the memory 
salesperson’s art, while all the jobs lie quiescent 
awaiting the pleasure of the Resource Manager or 
some other such system magus. 

(b) Saturation 

The obfuscatory nature of “saturation” lies in the 
fact that saturation is not a measure but a binary 
condition: the change in the quality of a service 
which moves from an unsaturated condition to a 
saturated one is an abrupt discontinuity: service 
effectively stops and the input queue becomes in¬ 
finite. References to “80 percent of saturation” 
thus really mean “80 percent of capacity”, and are 
doubly obfuscatory because saturation can be 
largely independent of capacity. We can best illus¬ 
trate this by means of an example: Consider a long, 
narrow footbridge with a capacity of 2000 pounds. 
Thirty-nine small boys, each weighing less than 50 
pounds, would not exhaust its capacity but would 
surely saturate it (they wouldn’t all fit at once) for 
quite a while. A single man leading an elephant, on 
the other hand, would not saturate the bridge but 
would exceed its capacity (see Exhibit IV). 

A useful related obfuscation is the measurement 
of throughput as a percentage of capacity, for many 
people ignore the basic fact that “capacity” 
changes with workload and environment. They 
consider “capacity” a configuration constant, de¬ 
spite the fact that they know full well that any 
reasonable multiprogramming system has a smaller 
capacity when restricted to highly compute-bound 
jobs than when fed a mixture of compute- and I/O- 
bound work. This blind spot can be exploited in 
other ways; it is much less well-known, for exam¬ 
ple, that any multiprogramming system strongly 


dominated by priority considerations has a smaller 
capacity than a system free to assign requested 
resources (such as the CPU) in an optimal fashion. 5 
A word to the wise, one hopes, is sufficient. . . . 
(c) Lines of Code 

This measure, being directed at human productiv¬ 
ity, might be considered by some to be somewhat 
outside the scope of this paper, but programmer 
performance is an element of computer center per¬ 
formance and lines-of-code is superbly obfusca¬ 
tory. More importantly, lines-of-code is perhaps 
the best embodiment of one of the great philosoph¬ 
ical rocks upon which Obfuscatory Measurement 
is founded: 

It is nearly always possible to substitute 

numbers for judgment. 

A timid person might hesitate to use lines-of-code 
on the grounds that it is patently absurd (is the 
Beer Bottle Song [“One hundred bottles of beer on 
the wall. . . .”] better than a Shakesphere sonnet? 
a limerick than a haiku? this paper than the Get¬ 
tysburg Address?), inasmuch as it ignores quality. 
Such a person severely underestimates the power 
of numbers to convince and confuse ... in a word, 
to obfuscate. 


L’ENVOI 

It must, alas, be admitted that an occasional obfuscator is 
taken to task for an excess of creative zeal. Should that 
unhappy fate befall you I can offer no defense better than 
that provided by Pooh Bah, who under such like circumstan¬ 
ces, 6 claimed that his obfuscation was “merely corrobora¬ 
tive detail, intended to add artistic verisimilitude to an oth¬ 
erwise bald and unconvincing narrative.” 
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INTRODUCTION 

Most businesses view the data processing staff as a central¬ 
ized resource, providing a specialized service to all segments 
of an organization. Centralization has occurred in response 
to the instrinsic cost and expertise benefits available through 
a concentration on this discipline. The inevitable separation 
from the client organization results in the establishment of 
a consultant/client relationship, e.g., the personnel respon¬ 
sible to support and accomplish the organization’s opera¬ 
tional objectives reside in a separately managed and con¬ 
trolled area. This presents a significant problem—one which 
often results in the operational area being accountable for 
an objective, while the data processor holds the key to its 
accomplishment. The data processor, who is charged with 
maintaining a high level of technical data processing com¬ 
petence along with developing an understanding of the 
clients objectives must clearly understand his position. On 
the other hand, the client, who has ultimate accountability 
for results, often has little, if any, data processing knowledge 
or ability to deal effectively with the data processor. 

Given this framework, how do data processor and client 
personnel communicate effectively? Can the data proc¬ 
essing systems developed perform as required? Can 
specific performance measurement criteria be developed? 

A plan must be developed to help both the data processor 
and client address these questions. The basic principles of 
the plan must be simple and capable of application across 
a wide range of client requirements. 

STANDARDS OF PERFORMANCE TECHNIQUE 

Given a corporation’s major investment in data processing 
systems and resources, an effective process to ensure re¬ 
quired human intervention can consist of the following steps: 

• Establish “Standards of Performance” for each data 
processing system 

• Measure the system’s performance as it compares with 
each standard each month. 

• Recognize and reward the performance contributions of 
both data processor and client personnel. 


In labor-intensive corporations, effective expense control 
programs must concentrate on salary costs and, in turn, on 
the number of personnel required in a support role. This is 
the critical controllable expense. 

I have found that where “Standards of Performance” are 
instituted, people are motivated to produce more than the 
results desired. The data processor and client begin to com¬ 
municate effectively. And why not? For the first time they 
have a common base—a commonly understood and held set 
of measurement criteria. 

ILLUSTRATIONS 

A case in point: One of the larger data processing systems 
I have personally been associated with was an employee 
human resource data bank which supported both Personnel 
and Payroll functions of one of the largest multiple-line in¬ 
surance companys in the United States. The system accom¬ 
modates the personnel/payroll records of an active staff of 
over 30,000 employees throughout the United States, Can¬ 
ada, and Puerto Rico. 

In the course of a year, the system manipulates, calculates 
and processes in excess of 1.2 million transactions. Asso¬ 
ciated with this is the annual production and distribution of 
over 350,000 personnel reports and 860,000 paychecks. 

An integral part of the system was a client staff of 39 
employees who were charged with the specific responsibility 
of inputting, pre-editing, coding, balancing, researching, 
analyzing, distributing and controlling this employee data. 
A staff of 19 data processors were required to support the 
technical aspects of the system. (Exhibit I illustrates the 
relationships.) The “system” in a real sense consists not 
only of the data processing equipment, but the manual sup¬ 
port functions that people must perform. 

Two years after installation, the system was perceived to 
be performing unfavorably. The data processors could not 
consistently satisfy the Payroll and Personnel client’s 
“needs.” Work was backlogged. Flexibility to handle new 
demands was limited. Processing costs were high—service 
levels, poor. Problems of this nature continued in spite of 
a 33 percent addition to staff over a two year period. The 
problem diagnosed was a lack of advanced data processing 
technology. 

At this point, a Project Team, composed of data process- 
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A longer term benefit was shown when two years after 
performance standards were implemented, a 50 percent re¬ 
duction in data processor and a 10 percent reduction in client 
resources were achieved as a direct result of improved com¬ 
munications and working relationships. Clearly, the catalyst 
was an agreed upon set of performance standards. (Exhibit 
II illustrates the results.) 

I don’t pretend to suggest that all data processing prob¬ 
lems are the result of erroneous perceptions or that they can 
be readily solved. Another case in point—performance 
standards were developed which, when applied, soon high¬ 
lighted the fact that results were less than desired. An in 
depth probe revealed: 


Exhibit I—Work relationships before standards of performance 


ing and client personnel, was commissioned to investigate 
the feasibility of installing a more sophisticated technology. 
The first step taken by the team was to develop a set of 
objective criteria, against which system performance could 
be measured. These were termed “Standards of Perform¬ 
ance.” Using them as a basis, the existing technology was 
evaluated. To everyone’s surprise, the existing system met 
the desired performance levels. In short, while the percep¬ 
tion of poor performance existed, when evaluated against 
specific, desired criteria, we were able to demonstrate that 
the system was performing satisfactorily. 

Unfortunately, as is often the case, it’s the perceptions 
that people have that counts. Our real problem therefore, 
was one of education—to correct an erroneous judgment. 
The culprit was identified as a real lack of understanding of 
system requirements and capabilities from both the data 
processor and client perspectives. The data processor was 
expending a large proportion of effort on activities that were 
not critical to the client personnel’s operations. In turn, the 
clients were not communicating their essential needs to the 
data processor. To further compound the problem, in the 
final analysis, the Human Resource System was in fact ad¬ 
equately servicing the end users and was more than meeting 
their expectations. In the course of the project team’s inter¬ 
view process all end users contacted were in agreement on 
this point. In spite of a constantly increasing input volume 
the system continued to produce secure, controlled, timely, 
and accurate output. 

The Project Team recommended that an “Operational 
Improvement” was warranted rather than installing a new 
data processing technology. That is, all manual input han¬ 
dling activities be combined into a single unit, with efficient 
workflow and procedures. Work measurement controls were 
installed to insure high productivity levels. In addition, the 
team recommended maintaining a strict adherence to the 
Human Resource System’s “Standards of Performance.” 

The recommendations were accepted by management and 
vigorously implemented. Immediate cost avoidance stem¬ 
ming from cancellation of the projected data processing de¬ 
velopmental effort was $200,000. More importantly, how¬ 
ever, system performance was perceived to be and was in 
fact meeting specific expectations! 


• Standards were ignored. While performance criteria 
were sound, they were ignored or inconsistently ap¬ 
plied. 

• Management placed little emphasis on monitoring ac¬ 
tual results. This was visible to the employees since 
normal supervisor recordkeeping was delegated down¬ 
ward. 

• Results were not reviewed between the data processor 
and client personnel. No ongoing constructive dialogue 
existed. 

When the results of the review were presented to man¬ 
agement they were accepted and constructively addressed. 
Within three months performance had returned to satisfac¬ 
tory levels. Again this was accomplished without the need 
to launch a new data processing system’s development ef¬ 
fort. 

PERFORMANCE MEASUREMENT BENEFITS 

When “Standards of Performance” are installed in an 
area, their impact while subtle, is dramatic in terms of re¬ 
sults. For the first time the data processor and client have 
the same reliable and objective criteria on which to evaluate 
their collective performance. Usually some disturbing facts 
surface: 



Exhibit II—Work relationships after standards of performance 
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• The data processing supervisor is not normally allocat¬ 
ing his personnel to the critical client tasks. Too much 
attention is focused on mechanical efficiency. 

• Advancements in technology are not usually an answer 
to performance problems. 

• The client realizes he has not effectively utilized his 
available data processing resource. This realization can 
be a determining factor in greatly extending the life 
span of an existing data processing system, while at the 
same time realizing an attendant increase in perform¬ 
ance. The cost avoidance, associated with not devel¬ 
oping replacement systems is almost always substantial. 

Once the standards are in place, communication between 
client and data processor and repetitive performance feed¬ 
back is assured. This dialogue is an essential product of the 
“Standards of Performance” technique. It is the key to 
developing a balanced, factual perspective to data process¬ 
ing evaluation. 

Positive results are visible in terms of reduced overtime 
and work backlogs. When standards are used, effective 
client/data processor communications are established. There 
is almost always a significant reduction in data processor 
resources required. Individual and group performance can 
be judged based on objective measurable results. Managers 
are in a better position to recognize and reward performance 
(or lack thereof). The net impact is a reduction in cost for 
salary expense, while, at the same time realizing an improve¬ 
ment in system performance. 


STANDARDS DEVELOPMENT APPROACH 

The approach employed to arrive at a data processing 
system’s “Standards of Performance” is based on applying 
the management principles taught in the Louis A. Allen 
Project Seminar (Management Planning and Control) to the 
data processing environment. 

“—ESTABLISH THE KEY OBJECTIVE: Identify the 
key data processing environment objective. 

—ESTABLISH CRITICAL OBJECTIVES: Statements 
of the most important (continuing) results which must 
be accomplished if the key objective is to be realized. 

—DEVELOP STANDARDS: Standards developed for 
each critical objective establish the criteria for effec¬ 
tive performance. ” 1 

The crucial management ingredient underlying this approach 
is the need to reach understanding and agreement between 
data processor and client. 


STANDARDS DEVELOPMENT ILLUSTRATION 

“Standards of Performance” for a hypothetical career 
management system could be constructed as follows: 


KEY OBJECTIVE 

To provide a reporting framework which supports the de¬ 
velopment, installation, and maintenance of personnel ca¬ 
reer management programs within the corporation. Further, 
to be responsive to report users’ status and information 
needs. 

CRITICAL OBJECTIVE 1 

To provide calculation and report development flexibility so 
that personnel career management reports can be tailored to 
report user needs on a timely, economical basis. 

STANDARDS 

• To develop new reports in not more than 34 elapsed 
days. 

• To modify existing reports in not more than 22 elapsed 
days. 

• To enable client report generation in not more than 10 
elapsed days. 

• To set-up a new employee “On File” in not more than 
three elapsed days. 

• To revise input forms in not more than 11 elapsed days. 

Report development performance measures are intended 
to eliminate the inefficient usage of reports. Emphasis is 
placed on delivering exception reports and summary level 
reports rapidly. Flexibility must be developed so that report 
users do not feel a need to request voluminous detailed 
reports. This need is normally generated in reaction to an 
unresponsive or a costly developmental process. 

Specific measurement criteria can be visualized in terms 
of a two dimensional matrix. (Exhibit III illustrates sample 
relationships). Procedurally, this matrix can be developed 
in the following three steps: 

STEP 1— REPORT REQUIREMENTS are analyzed in 
terms of critical client needs. Report categories must be 
clearly defined and capable of being measured. Addition¬ 
ally, the data processor and client must be in complete 
agreement with the categories selected. 

STEP 2— FUNCTIONS that must be performed, to insure 
error-free reports, are documented for all participants. 
Typically this will include report recipients in addition to 
the data processor and client. It is usually found that lines 
of responsibility are not clearly defined, which causes 
inefficiencies and duplication of effort. 

STEP 3— ELAPSED TIMES must be assigned for each 
REPORT REQUIREMENT and will reflect past perform¬ 
ance and future requirements. Times must be reasonable, 
attainable and should reflect a common, understood, and 
agreed to goal. 

The end result of setting report development measurement 
criteria will be a common understanding of the report de¬ 
velopment process for both the client and data processor, 
along with supporting a framework for client communica¬ 
tions with report requestors. The client can now pursue, on 
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r REPORT REQUIREMENTS 

Function 

Develop 

New 

Reports 

Modify 

Existing 

Reports 

Client 

Report 

Generation 

New 

Employee 

Set-Up 

Revision 
of Input 
Forms 

Initial Report User Contact 

1* 

1 

1 

1 

1 

Finalize Specifications 
Report User & Client 

1 

1 

1 

2 

10 

Finalize Specifications 
Client & Data Processor 

5 

3 

- 

- 

- 

Analysis & Costing Client 
& Data Processor 

5 

3 

- 

- 

- 

Programming 

10 

5 

2 

- 

- 

Unit Test Data Generation 
(Client) 

5 

3 

3 

- 

- 

Unit Testing 

3 

2 

2 

- 

- 

Verify Results (Client) 

1 

1 

1 

- 

- 

Report Acceptance (Data 
Processor) 

3 

3 

- 

- 

- 

TOTALS 

34 

i 

22 

i 

10 

3 

11 


* All time estimates are in terms of elapsed days. 

Exhibit III—Personnel career management report matrix 


a factual basis, report delivery dates and can advise on the 
risks involved in attempting to unreasonably accelerate de¬ 
livery dates. 

CRITICAL OBJECTIVE 2 

Complete data processing production runs accurately and 
on schedule. 

STANDARDS 

• 95 percent of all production runs will be completed 
within five (5) working days from receipt of the latest 
input item or input cut off date. This will apply unless 
other specific schedules are established. 

• Machine resources will not exceed $5,000 monthly. 

These measurement criteria are intended to be applied after 
the initial report development process is completed. The 
measurements ensure that recurring reports are delivered on 
a timely basis—it makes no sense to develop a report in a 


rapid and cost effective manner and then not monitor the 
recurring delivery process. 

CRITICAL OBJECTIVE 3 

Implement minor report changes (1-10 days) on a timely 
basis. 

STANDARDS 

• Resolve 98 percent of priority “A” changes prior to the 
next update (unless other arrangements agreeable to 
client and data processor have been made). 

• All changes will be prioritized through the combined 
efforts of client and data processor personnel. Adding 
changes after the monthly schedule is established will 
result in a re-prioritization of existing planned work. 
Monthly planning of this activity should result in 95 
percent of scheduled changes being completed without 
change. 

• Implement 95 percent of all priority “B” (Balancing 
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problems, report format/content changes) changes and 
90 percent of all priority “C” (noncritical) changes on 
a monthly basis as reflected on the agreed-upon 
monthly maintenance schedule. 

It must be noted that for proper application of these stand¬ 
ards the prioritization process must be agreed to and under¬ 
stood by both client and data processor. The priority process 
is negotiable, however typical priority definitions are as 
follows: 

A. Essential to next production processing cycle. 

B. Desirable change—but not essential to next cycle 
process. 

C. Changes to improve system performance, control, 
input and report processing. 

Once the reports are developed and the ongoing data 
processing system is installed, requirements for system 
change occur. The origin of change is complex and can be 
generated from report user, client, or data processor per¬ 
sonnel. However generated, change requirements must be 
controlled in a manner that identifies the most critical re¬ 
quirements. The intent is to meet maintenance measurement 
criteria while minimizing client and data processor staffing 
needs. 

DEVELOPMENT TIMING 

Defining a system’s standard of performance benefits both 
the client and the data processor. Ideally these performance 
criteria should be established early in a data processing 
system’s developmental process. The standards ensure that 
the client fully communicates critical needs. Since the stand¬ 
ards are normally given in terms of response times (elapsed 
days) to perform an activity, the data processor has the 
necessary information to answer difficult system design 
questions—while fulfilling customer requirements with a 
minimum investment of data processor resources. 

The standards of performance can also be developed for 
systems already installed. Typically maintenance activities 
can generate an unacceptable level of resource commitment 
in response to undefined but assumed objectives. Again, the 
standards can document the clients critical needs and can 
serve as a vehicle to evaluate an existing system’s perform¬ 
ance. Closely monitored, it can also indicate a need to up¬ 
grade or replace the existing system. 

INGREDIENTS FOR SUCCESS 

The following four elements are necessary to ensure the 
success of the “Standards of Performance” technique. 

STEP 1—ESTABLISH STANDARDS OF PERFORM¬ 
ANCE: The critical activity is to first establish the stand¬ 
ards. Without sound standards the technique will fail. 

These performance standards cannot normally be estab¬ 


lished by methods such as time ladder studies, random 
sampling, stopwatch techniques, or modal time analysis. 
In other words the type of standards we are dealing with 
are not translatable to engineered time standards; reli¬ 
ance is placed on a combination of historical trends and 
consensus agreement. 

STEP 2—PURSUE DATA PROCESSOR AND CLIENT 
UNDERSTANDING AND AGREEMENT: Having de¬ 
veloped the standards, we must now set agreed to elapsed 
times and arrive at a complete understanding of require¬ 
ments in both data processor and client areas. This must 
be done at the highest management levels in both areas. 
Again, without complete understanding and agreement, 
the chances of success are reduced. 

STEP 3—REGULARLY MONITOR THE PERFORM¬ 
ANCE CRITERIA: The performance results must be dis¬ 
cussed and acted upon on a monthly basis. This dialogue 
is yet another ingredient in ensuring the success of the 
technique since most standards cannot be as precise as an 
engineered time standard. A continuing reevaluation is 
the method used to test the standards as initially estab¬ 
lished and to answer such questions as: ARE RESULTS 
MEETING STANDARDS? IF NOT, WHY NOT? 

STEP 4—REFINE STANDARDS WHEN NECES¬ 
SARY: Given the imprecise nature of the standards, we 
must be willing to reevaluate them. Many external factors, 
such as client reorganizations or end user requirement 
changes can render some standards obsolete. Without 
continuous evaluation and updating all gains initially made 
can be lost. 

CONCLUSION 

No data processing effort should be attempted without first 
establishing “Standards of Performance.” Differing client 
requirements and objectives may require a unique set of 
performance criteria, however, there is no doubt that meas¬ 
urement criteria are mandatory. 

The challenge to improve is a shared responsibility. Here 
is a new technique which, if properly applied, will result in 
high “system productivity” and the achievement of clearly 
defined levels of performance with communications which 
effectively link all parties of “The System.” Remember 
“The System” consists of automatic data processing equip¬ 
ment plus the personnel that make it go and Perceptions. 
While it is important to understand and control the people/ 
machine relationship, it is equally as important that client 
and data processor share a common perception of perform¬ 
ance. 
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Effects of peripheral processor wait list 
positioning on system performance 

by RONNIE G. WARD, BECKY B. TURNER and GALEYN JOE HUBBARD 

University of Texas at Arlington 
Arlington, Texas 


INTRODUCTION 

Strauss 1 developed a simple analytic model of the HASP 
Execution Task Monitor (HETM) for use in studying per¬ 
formance improvements that result from HETM’s applica¬ 
tion and whether or not HETM should be modified to con¬ 
sider different memory speeds of OS/MFT partitions at a 
specific installation. The cyclic-server multiprogramming 
model he developed is demonstrably valid. 2 That is, using 
service rates fitted to the model, predicted results approxi¬ 
mate measured results reasonably well, and one may have 
confidence that the model accurately reflects system per¬ 
formance. 

In IBM’s OS, one or more job classes are assigned to 
each system initiator program, which selects jobs to intro¬ 
duce into the system on a priority basis within class. OS 
uses preemptive priority CPU task dispatching. Tasks are 
represented to the operating system using a task control 
block (TCB). The set of TCB’s corresponding to active tasks 
are chained in priority order (highest to lowest). The dis¬ 
patcher assigns the CPU to the first ready task found by 
scanning the TCB chain in priority order. An event comple¬ 
tion that causes a higher priority task to become ready 
generates a task switch and a running lower priority task 
will be preempted. HETM regularly examines the CPU uti¬ 
lization of OS tasks and rearranges the TCB chain in an 
attempt to more equitably distribute CPU service. As 
Strauss points out, the effect is to keep I/O bound tasks 
active without unduly harming service to CPU bound tasks. 

When an executing program solicits an I/O operation, the 
I/O supervisor 3 associates a request element (RE) with the 
intended work. If the I/O operation cannot be started im¬ 
mediately, the RE is positioned in a wait list associated with 
the addressed device. An installation may specify at system 
generation time the wait list positioning policy to be used 
for each device. For example, an installation may choose 
for DASD devices either first-come first-served (FCFS) 
queuing, dispatching priority positioning, or seek address 
positioning. 

The purpose of this paper is to quantify in terms of the 
results of an analytic model the effect wait list positioning 
can have on system performance. In systems where CPU or 
channel time is a bottleneck, the model can be used to study 


the importance of wait list positioning in increasing or de¬ 
creasing job class response. The analytic model of different 
wait list positioning strategies is first presented. The HETM 
stability condition given in Reference 1 is then re-examined. 
Finally, performance predictions of the positioning policies 
are discussed and related to system and job class perform¬ 
ance". 


THE ANALYTIC MODEL 

Figure 1 depicts a simple cyclic exponential server model 
that is a closed network permitting no exogenous arrivals or 
departures. There are three task classes a, /3, and y specified 
by their CPU service rates /t* and peripheral service rates 
St where i is a, /3, or y. The characters a, b, and c denote 
the tasks ranked by preemptive CPU priority; task a can 
preempt b which can preempt c for CPU service. A mapping 
is defined as a permutation of the a, 13, y tasks onto the 
priority designations a, b, c. It is the HETM reordering of 
the OS TCB chain that determines a particular mapping. 
The tasks a, b, and c are serviced by an exponential process 
CPU server with mean rate /*,. Following CPU service, the 
task enters a peripheral processor (PP) wait list and even¬ 
tually receives exponential process peripheral service with 
mean rate 8*. After PP service, the task may enter the CPU 
queue or be serviced by the CPU. The tasks cycle contin¬ 
uously through a CPU-PP service loop. 

The system is modeled in steady state during HASP in¬ 
tervals between changes to the TCB chain. Analytic and 
simulation studies have shown this to be a reasonable as¬ 
sumption. 1 

In the model, each of the tasks can be in any one of the 
following five places in the system: 

0. Waiting for CPU service. 

1. Obtaining CPU service. 

2. At the end of the PP wait list. 

3. At the head of the PP wait list. 

4. Obtaining PP service. 

The state of the system is completely specified by the con¬ 
tents of four of the five positions. A four component state 
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Figure 1—Cyclic exponential server model 


vector v is used to denote the system state where the 7th 
component of v denotes the yth position of the system and 
the contents of each component can be a, b, c, or x (x 
denotes an empty position). For example, the state v=axbc 
indicates that a “class a” task is receiving cpu service, a 
“class b" task is at the head of the I/O queue, a “class c” 
task is receiving PP service, and the second position in the 
PP wait list is unoccupied. 

Using this notation, the possible state vectors v k of five 
PP wait list positioning strategies are written as follows: 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

{ abcaxcaaxbxbxaxx 
xxxxcxxxcxaxbxba 
xxbxbacxaccacbab 
xaababbcbabcaccc 

FCFS and RANDOM positioning (k= 1) 

I'abcaxcaxxbx 
Vi 2 = Jxxxxbxxabxa 
isfeii Ixxbxcabcaab 
(xaababcbccc 

Last-come First-served (LCFS) (k= 2) 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

{ abcxaacxabbx 
xxxcxxxcxxxb 
xxbbxcaaxcaa 
xaaabbbbcacc 

Dispatching Priority Positioning (k= 3) 

{ abcxaacxxb 
xxxbxxxaax 
xxbcxbabca 
xaaabcbcbc 

Inverse Dispatching Priority Positioning {k= 4) 

The application of a wait list positioning policy other than 
FCFS is quickly observed by studying the state transitions. 
For example, u 6 3 -»t; 8 3 since the a task would position ahead 
of the c task after completing CPU service. On the other 
hand, i> 7 4 -»y 9 4 since an inverse priority scheme is used. 

As pointed out in Reference 4, FCFS scheduling is an 


appropriate model of secondary storage I/O devices because 
preemptive scheduling is usually not possible or efficient. 
However, given that the device is not preemptable, task 
positioning in the device wait list can still influence the 
performance measures for an individual job class. Typically, 
analytic models presented in papers such as in References 
5 and 6 assume that device scheduling is independent of the 
processes generating the requests. However, this may not 
always be the case as already discussed in the instance of 
OS. 3 With the above models, it is possible to determine 
analytically the impact of wait list positioning on job class 
performance. Note that LCFS scheduling is feasible only 
for inteijob requests. 

Each state vf has an associated steady state probability 
P Vik . These equilibrium probabilities can be determined by 
solving the system balance equations [1,4]. That is, for all 
states Vi k , 

X P Vj k x (rate of flow from v / to v { k ) 

j 

=Pv t kX (rate of flow out of v t k ) 

From these probabilities, the task CPU utilization, PP uti¬ 
lization, queue time, and cycling rates may be determined. 1,2 
Also note that the distinction between FCFS and random 
positioning is in the state transitions and balance equations. 
For example, under FCFS, the c task exiting the CPU results 
in However, under random positioning we could 

also have the transition iV-^13 1 with an equal probability 
(.5). The flow into certain states is therefore altered. For 
instance, the balance equation for v 5 x under FCFS is 

Pxcba Pc P C xba 

and under random positioning we would have 
^ a Pxcba • 5 Pc Pcxba + .5 p h P b 

xca 

Another interesting model result aside from service center 
utilizations and task throughputs is the task switching rate 
TSW fc . A task switch occurs when a higher priority task 
becomes ready to use the CPU. For the above models, we 
have for TSW 1 and TSW 3 : 

(F*bxxa F Pcxba F Pbxca ) F Sb Pcxa b 

and for TSW 2 and TSW 4 : 

ba {P bxxa F .Pcxba ) F 8b Pcxab 

Note that although the method of calculation may be the 
same, the results will differ because of different valued state 
probabilities. The TSW fc give an indication of the operating 
system overhead due to task switching. Naturally, the lower 
the value the lower the overhead. Similar overhead meas¬ 
ures can be obtained for wait list positioning using a priority 
scheme. 


HETM STABILITY CONDITION 

At each HASP interval, HETM rearranges the subset of 
the OS dispatching chain constituted by an installation spec- 
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ified dynamic priority group. Tasks are ranked in inverse 
order of their CPU utilization histories. Equation (1) is used 
to determine the CPU utilization history h t>n of the nth task 
at HASP interval t. 

ht,n = cpu^ n + ht—i >n Hf / N (1) 

N is the number of tasks in the dynamic priority group, 
cpu f>n is the CPU counts observed for the nth task during 
the rth HASP interval, and if<=2|L 1 (cpu w +/i ( _i, f ). 

Adopting the notational conventions of Reference 2: 



~K~ 


"cpu <>a " 

bt= 

ht,p 

, c t = 

cpu t,ts 


ht,y 


CPU uy 


equation (1) can be rewritten for the three job model as: 


h t , tS3 t. Define bt~ht' if and only if s 1 = s 1 ', s 2 — s 2 \ and 
s 3 = s 3 '. Then ~ is an equivalence relation. 

Let E be the equivalence class formed by Ji^-dr (i.e. if 
W={h t | t>0},then E={h t eW \ h (rn - 1)R ~h t }). Deriving a sta¬ 
bility condition requires that h mR —h (m - m . In order to have 
h mR eE, we must have some permutation s t , s 2 , ^3 of a, (3, 
y such that 

hmR.Sj — hmR,S2 — hmR,s 3 ( 8 ) 

and 

— ^(m-1 )R,s 2 — h(m-l)R,s 3 (9) 

Thus, if we let k t , k 2 , k 3 be the permutation of 1, 2, 3 for 
the mapping s lt s 2 , s 3 corresponding to the a, /3, y index of 
h t , from (8) we get: 

A kl *C R +h (m _ 1)RiSl <A k2 *C R (10) 

"h h(m— l)B,S 2 — Afcg* C ~^b(m~l)R,s 3 


where 


bt—A(h t -i+C t ) 



Strauss 1 gives the solution to (3) as: 


(3) 


bt=A 



For a simple stable mapping (i.e. the mapping for the next 
HASP interval is the same as the previous) with h o =0, 
equation (4) reduces to 

bt=tAC (5) 

where C is the CPU utilization vector associated with the 
repeating mapping. 

For higher order repeating mapping patterns, Strauss 1 de¬ 
termines that if h t is to repeat every /?>1 control intervals, 
then h mR must equal h nR for all integers m and n. Thus from 
(4): 

( mR , 

2 Uj| "b h(m-l)R (6) 

j=(m-l)R+l / 

Clearly, for h mR to equal h (m - 1)R (and thus repeat the map¬ 
ping pattern), we must have 

( mR v 

2 Cj) = o. 

j=(m-l)R+l / 

Since A is idempotent, the vector C R of (7) must have equal 
components. 


C R = 


CpUo^l R 

cp u/ = 2 Cj 
cpu 7 B j=1 


(7) 


To relax the requirement that C R have equal components 
consider the following. Let s lf s 2 , s 3 and s t \ s 2 ', s 3 ' be 
mutations of a, (3, y such that the mappings generated by h t 
and h t , imply h Ul <h tiS2 <h Us and h t , >Sl ,<h v>S2 ,< 


where A t * denotes the ith row vector of A. Using (9), we 
may guarantee (10) if 

A kl *C R ^A k2 *C R ^A k *C R . 

Adding 1 /3(cpu« iJ +cpu /3 i? -l-cpu 7 /r ) to each term yields 
cpu/<cpu S2 * <cpu S3 *. 

We assume in the above discussion that equal history values 
imply that a precedes (3 which precedes y. 

PERFORMANCE COMPARISONS 

This section discusses the relative performances of the 
five wait list positioning policies. The models presented in 
an earlier section were solved for the two sets of and 8, 
presented in Table I. 1 * 2 These parameters are both interest¬ 
ing and representative. The ju, f and 8 t of Example 1 were 
derived by fitting the model to an OS/MFT system operating 
under HASP. Example 2 parameters are presented in Ref¬ 
erence 1 and it can be seen that a is a light CPU, heavy I/O 
task; f3 is a moderate CPU, light I/O task; and y is a heavy 
CPU, moderate I/O task. It should be noted that since the 
balance equations are homogeneous, a given set of utiliza¬ 
tions can result from an infinite set of service rates. The 
relative values of the service rates are therefore important, 
and representative values such as those in Example 2 may 
be used. 

The model predictions for all possible mappings using the 
parameters of Table I are presented in Tables II-VI for 
Example 1, and Tables VII-XI for Example 2. Using the 
derived stability condition, the different wait list positioning 
models were examined for stable mapping patterns through 
simple simulation. For Example 1, #4 appears to be a first 
order stable mapping when limiting on the history values 1 is 
imposed. For Example 2, #1 is a first order stable mapping. 
The model predictions are compared using these two stable 
mappings. 

Clearly, with both sets of data and under all positioning 
policies, total PP utilization does not significantly vary. This 
results from the PP device not being preemptable. However, 
total CPU utilization fluctuates significantly, most noticea- 
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TABLE I.—Model Parameters 

Example 1 Example 2 


Task 

CPU Service 

PP Service 

CPU Service 

PP Service 


y. 

6. 

y. 

<$. 


l 

l 

i 

i 


, -1. 

-1 

-1 

-1 


(sec ) 

(sec ) 

(sec ) 

(sec ) 

a 

6 

11.94 

25 

5 

3 

99.94 

18.43 

15 

75 

Y 

81 

24.48 

5 

25 


TABLE II.—Priority CPU Scheduling with FCFS PP Scheduling 


Mapping CPU Utilization PP Utilization Cycling Rates 


No. 

abc 

a 

b 

c 

E cpu 

a 

b 

c 

E pp 

System 

Total 

TSW 

a b 

c 

1 . 

a3y 

.605 

.033 

.036 

.674 

.304 

.181 

.120 

.605 

1.279 

.12 

.60 .56 

.49 

2. 

ay 3 

.606 

.041 

.029 

.676 

.304 

.134 

.160 

.599 

1.275 

.12 

.61 .55 

.49 

3. 

3ay 

.104 

.490 

.035 

.629 

.566 

.246 

.115 

.927 

1.556 

1.35 

1.74 .49 

.47 

4. 

3ya 

.086 

.090 

.422 

.599 

.468 

.298 

.212 

.978 

1.577 

1.67 

1.44 1.22 

.42 

5. 

Y3a 

.109 

.074 

.419 

.602 

.362 

.400 

.211 

.972 

1.574 

1.74 

1.48 1.23 

.42 

6. 

ya3 

.150 

.482 

.028 

.659 

.496 

.242 

.150 

.888 

1.548 

1.65 

2.03 .48 

.46 


TABLE III.—Priority CPU Scheduling with Priority PP Scheduling 


Mapping 

CPU Utilization 

PP Utilization 



Cycling Rates 










System 




No. 

abc 

a 

b 

c 

E cpu a 

b 

c 

E pp 

Total 

TSW 

a b 

c 

1 . 

a8Y 

.623 

.034 

.029 

.686 .313 

.183 

.097 

.594 

1.280 

. 12 

.62 .56 

.40 

2. 

ay 3 

.625 

.041 

.025 

.692 .314 

.136 

.137 

.587 

1.279 

.12 

.63 .56 

.42 

3. 

3ay 

.110 

.494 

.028 

.631 .596 

.248 

.091 

.935 

1.565 

1.42 

1.83 .49 

.37 

4. 

3ya 

.095 

.099 

.279 

.472 .517 

.327 

.140 

.984 

1.456 

1.20 

1.59 1.33 

.28 

5. 

Y3a 

.121 

.081 

.277 

.478 .399 

.440 

.139 

.979 

1.457 

1.28 

1.63 1.35 

.28 

6. 

ya3 

.159 

.485 

.023 

.667 .526 

.244 

.126 

.895 

1.562 

1.75 

2.15 .48 

.39 


TABLE IV.—Priority CPU Scheduling with Inverse Priority PP Scheduling 


Mapping CPU Utilization 

No. abc abc E cpu 

PP Utilization 

a b c E pp 

System 

rotai 

TSW 

Cycling Rates 

a b c 

1 . 

a3y .488 .045 .057 

.589 

.245 

.243 

.188 

.676 

1.265 

.20 

.49 .75 

.77 

2. 

ay3 .490 .054 .045 

.589 

.246 

.179 

.245 

.670 

1.258 

.20 

.49 .73 

.75 

3. 

3ay .095 .468 .053 

.616 

.517 

.235 

.175 

.927 

1.543 

1.33 

1.59 .47 

.71 

4, 

3ya .079 .093 .478 

.650 

.427 

.308 

.240 

.976 

1.625 

1.93 

1.31 1.26 

.48 

5. 

y3a .095 .076 .477 

.649 

.314 

.416 

.240 

.970 

1.618 

1.92 

1.28 1.28 

.48 

6. 

ya3 .134 .452 .040 

.627 

.444 

.227 

.220 

.891 

1.518 

1.57 

1.81 .45 

.68 
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TABLE V.—Priority CPU Scheduling with Random PP Scheduling 

Mapping CPU Utilization PP Utilization Cycling Rates 

System 


No. 

abc 

a 

b 

c 

3 cpu 

a 

b 

c 

E pp 

Total 

TSW 

a b 

c 

1. 

aBy 

.610 

.032 

.036 

.678 

.306 

.176 

.119 

.602 

1.280 

.13 

.61 .54 

.49 

2. 

ay 3 

.610 

.039 

.030 

.679 

.306 

.130 

.161 

.597 

1.276 

.13 

.61 .53 

.49 

rr 

o. 

Bay 

.104 

.454 

.043 

.600 

.565 

.228 

.142 

.934 

1.535 

1.31 

1.73 .45 

.58 

4. 

3ya 

.088 

.087 

.420 

.595 

.481 

.286 

.211 

.978 

1.573 

1.67 

1.48 1.17 

.42 

5. 

y3a 

.112 

.072 

.420 

.603 

.371 

.390 

.211 

.972 

1.575 

1.74 

1.51 1.20 

.42 

6. 

ya3 

.148 

.445 

.034 

.627 

.489 

.224 

.185 

.898 

1.525 

1.60 

1.99 .45 

.57 


Mapping 

No. abc 

TABLE VI.—Priority CPU Scheduling with LCFS PP Scheduling 

CPU Utilization PP Utilization 

System 

a b c E cpu a b c E pp Total 

TSW 

Cycling Rates 

a b e 

1. 

a3y 

.549 .036 .050 

.635 .276 

.198 

.165 

.639 

1.274 

.17 

.55 .61 

.68 

2. 

ay 3 

.565 .040 .038 

.644 .284 

.133 

.207 

.624 

1.267 

.16 

.57 .54 

.63 

3. 

Bay 

.101 .335 .069 

.505 .549 

.168 

.229 

.946 

1.451 

1.06 

1.69 .33 

.93 

4. 

3ya 

.085 .082 .479 

.646 .465 

.271 

.240 

.976 

1.622 

1.87 

1.43 1.10 

.48 

5. 

Y8a 

.104 .071 .477 

.652 .343 

.386 

.240 

.969 

1.621 

1.93 

1.40 1.19 

.48 

6. 

ya3 

.139 .334 .053 

.526 .458 

.168 

.290 

.916 

1.442 

1.28 

1.87 .33 

.89 


TABLE VII.—Priority CPU Scheduling with FCFS PP Scheduling 


Mapping 

No. abc 

CPU Utilization PP Utilization 

a b c E cpu a b c E pp 

System 

Total 

TSW 

Cycling Rates 

a b c 

1 . 

aBy 

.159 .204 

.309 

.672 .793 .041 

.062 

.895 

1.567 

2.79 

3.96 

3.06 

1.55 

2. 

ay 3 

.158 .400 

.104 

.663 .789 .080 

.021 

.890 

1.553 

2.60 

3.95 

2.00 

1.57 

3. 

3ay 

.546 .087 

.180 

.813 .109 .434 

.036 

.579 

1.392 

8.65 

8.19 

2.17 

.90 

4. 

3ya 

.778 .171 

.013 

.961 .156 .034 

.067 

.256 

1.218 

11.57 

11.66 

.85 

.33 

5. 

y3a 

.759 .109 

.033 

.901 .152 .022 

.165 

.338 

1.239 

3.02 

3.80 

1.63 

.82 

6. 

ya3 

.683 .070 

.068 

.821 .137 .349 

.014 

.500 

1.320 

2.77 

3.41 

1.75 

1.02 





TABLE VIII, 

—Priority CPU Scheduling with Priority PP Scheduling 




Mapping 

CPU Utilization 

PP Utilization 



Cycling Rates 









System 




No. 

abc 

a b 

C 

E cpu 

a b 

c 

E.PP. 

Total 

TSW 

a 

b c 

1. 

aBy 

.160 .205 

.278 

.643 

.802 .041 

.056 

.898 

1.541 

2.63 

4.01 

3.07 1.39 

2. 

ay 6 

.159 .400 

.083 

.642 

.797 .080 

.017 

.893 

1.536 

2.49 

3.98 

2.00 1.24 

3. 

Bay 

.551 .087 

.161 

.799 

.110 .436 

.032 

.578 

1.377 

8.59 

8.26 

2.18 .80 

4. 

Byoi 

.782 .170 

.013 

.965 

.156 .034 

.065 

.256 

1.220 

11.73 

11.73 

.85 .33 

5. 

yBa 

.765 .108 

.032 

.905 

.153 .022 

.162 

.337 

1.242 

3.82 

3.82 

1.62 .81 

6. 

yaB 

.689 .069 

.055 

.813 

.138 .346 

.011 

.494 

1.307 

2.44 

3.44 

1.73 .82 
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TABLE IX.—Priority CPU Scheduling with Inverse Priority PP Scheduling 

Mapping CPU Utilization PP Utilization Cycling Rates 

System 


No. 

abc 

a 

b c 

Z cpu 

a 

b 

c 

app 

Total 

TSW 

a 

b 

c 

1 . 

a By 

.158 

.200 .313 

.671 

.791 

.040 

.063 

.894 

1.565 

3.72 

3.96 

3.00 

1.57 

2. 

ay 3 

.157 

.398 .128 

.683 

.785 

.080 

.026 

.890 

1.572 

3.20 

3.92 

1.99 

1.92 

3. 

Bay 

.491 

.092 .209 

.792 

.098 

.461 

.042 

.601 

1.394 

8.18 

7.37 

2.31 

1.04 

4. 

Bya 

.757 

.165 .016 

.938 

.151 

.033 

.082 

.266 

1.205 

11.41 

11.35 

.83 

.41 

5. 

yBa 

.748 

.100 .033 

.881 

.150 

.020 

.166 

.335 

1.216 

3.42 

3.74 

1.50 

.83 

6. 

yctB 

.653 

.073 .092 

.818 

.131 

.365 

.018 

.514 

1.332 

2.91 

3.26 

1.83 

1.38 






TABLE X.—Priority CPU Scheduling with Random PP Scheduling 





Mapping 

CPU Utilization PP Utilization 



Cycling Rates 









System 





No. 

abc 

a 

b c 

Z cpu a 

b 

c 

z pp 

Total 

TSW 

a 

b 

c 

1 . 

a By 

.160 

.161 .327 

.648 .799 

.032 

.065 

.896 

1.544 

2.97 

3.99 

2.41 

1.64 

2. 

ayB 

.159 

.360 .132 

.652 .795 

.072 

.026 

.893 

1.545 

2.90 

3.97 

1.80 

1.99 

3. 

Bay 

.494 

.092 .208 

.794 .099 

.461 

.042 

.601 

1.395 

8.22 

7.41 

2.30 

1.04 

4. 

Bya 

.765 

.159 .016 

.940 .153 

.032 

.079 

.264 

1.203 

11.43 

11.47 

.80 

.39 

5. 

yBa 

.758 

.085 .035 

.878 .152 

.017 

.173 

.342 

1.220 

3.02 

3.79 

1.28 

.87 

6. 

yaB 

.658 

.073 .092 

.822 .132 

.363 

.018 

.512 

1.334 

2.92 

3.29 

1.81 

1.37 


TABLE XI.- 

Mapping CPU Utilization 

No. abc a b c Z cpu 

—Priority CPU Scheduling with LCFS PP Scheduling 

PP Utilization 

System 

a b c Z pp Total 

TSW 

Cycling Rates 

a b c 

1 . 

aBy .159 

.193 .303 

.655 

.797 .039 

.061 

.896 

1.551 

3.05 

3.98 

2.90 

1.51 

2. 

ayB .159 

.383 .108 

.650 

.794 .077 

.022 

.892 

1.542 

2.72 

3.97 

1.92 

1.61 

3. 

Bay .531 

.089 .186 

.805 

.106 .445 

.037 

.588 

1.393 

8.47 

7.96 

2.22 

.93 

4. 

Bya .771 

.166 .015 

.951 

.154 .033 

.073 

.261 

1.212 

11.52 

11.57 

.83 

.37 

5. 

yBa .759 

.102 .033 

.894 

.152 .020 

.164 

.336 

1.230 

3.19 

3.80 

1.53 

.82 

6. 

yaB .687 

.069 .072 

.829 

.137 .347 

.014 

.499 

1.328 

2.77 

3.44 

1.74 

1.08 


bly in Example 1. For this case, the c task spends more time 
in the wait list under priority positioning and total system 
utilization is lowest, but throughput is maximized. Situated 
there, the c task cannot get to the CPU, thereby lowering 
total CPU utilization. Likewise, since both the a and b tasks 
position ahead of it in the wait list and given that their CPU 
use is comparatively light, their throughput jumps, raising 
overall system throughput. Obviously, a and b task turna¬ 
round time decreases while the c task’s increases. Further¬ 
more, task switching overhead drops under priority posi¬ 
tioning. Again, this occurs since the c task is in the PP wait 
list a larger percentage of the time. 

Over the range of (jli to S j( with equal job class charac¬ 


teristics, the a task will have better CPU and PP utilization 
under priority wait list positioning. This results in a lower 
response time and increased throughput. The same holds 
true for the c task under inverse priority positioning. Under 
priority positioning, the c task will always suffer greater 
response time and less throughput as does the a task under 
inverse positioning. This can be observed in Figures 2 and 
3. 

These results suggest a desirable scheduling strategy for 
either CPU bound systems, or more likely, batch systems 
supporting an interactive computing load. Faster turnaround 
will result for the high priority interactive requests at the 
expense of lower priority (batch) jobs. This is especially true 
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P 


. << 
l 


6 . 

l 



p. > 6 . 
l l 


P . >> 6 . 
l l 


Figure 2—CPU utilization with equal job class characteristics over the 
range of service rates 


for OS since it was not initially designed to support conver¬ 
sational systems. 

When priority wait list positioning is utilized under OS 
with HASP, a curiosity develops. At the time a task is 
created, OS assigns it a dispatching priority. At the Univer¬ 
sity of Texas Regional Computer Center all batch job tasks 
are created with the same dispatching priority. Since HETM 
does not alter the dispatching priority when it rearranges the 
TCB chain and the I/O Supervisor makes use of it to deter¬ 
mine queue position, then, in effect, random I/O scheduling 
occurs. Moreover, even if different priority values were 
assigned (or changed through the CHAP macro), I/O sched¬ 
uling would not always reflect HETM’s current arrangement 
of the TCB chain. In essence, this is a HETM shortcoming. 
The TCB chain ordering represents the most recent infor¬ 
mation regarding task characteristics, which are known to 
change dramatically during execution. By not updating dis¬ 
patching priorities to coincide with the TCB chain ordering, 
under priority wait list positioning HETM is ignoring an 
aspect of the problem it was intended to solve. 

With inverse dispatching priority positioning, it can be 


observed that neither a nor b task throughput is adversely 
affected. As one would expect, CPU utilization increases 
with this positioning policy. This results because the CPU 
bound job (a) is assigned to the c task. Given that PP use 
remains relatively unchanged, total system utilization will 
tend to maximize under this policy. This can be seen in the 
two examples. Also, observe that a more balanced cycling 
rate is a consequence. Inasmuch as the outcomes of this 
policy are not calamitous for the a task, systems with an 
under utilized CPU may find this scheduling strategy attrac¬ 
tive. The model also predicts, however, an increase in op¬ 
erating system overhead due to task switching. 

Though inferences concerning real system performance 
are being made, it would be inappropriate to cite percentage 
changes in the models’ predicted utilizations. This would 
attach an unsupported significance to the model results. This 
kind of observation can only be made in a specific situation 
using a perhaps more detailed model (one with additional PP 
devices and more job classes) with fitted service rates. Four 
job models were formulated and solved using representative 
service rates and the above observations remain true with 
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a,b,c - FCFS, a'.b'.c' - Priority, a , ' l b ,, ,c" - Inverse Priority 



u.<< 6. 6. y.> 6. u.>> 5. 

i i i x ii 

Figure 3—PP utilization with equal job class characteristics over the range 
of service rates 


marked changes in the percentage differences of the pre¬ 
dicted utilizations. The details of these models are not pre¬ 
sented. 


CONCLUSIONS 

Five peripheral processor wait list positioning policies are 
presented. Relative performances are compared using a sim¬ 
ple analytical model of a multiprogramming system with 
preemptive priority CPU dispatching on different classes of 
jobs. 

Using fitted 2 service rates, priority wait list positioning 
was found to maximize throughput while minimizing total 
system utilization. High priority jobs receive better through¬ 
put and decreased response time. On the other hand, inverse 
dispatching priority positioning tends to maximize total sys¬ 
tem utilization and is not ruinous to high priority job re¬ 
sponse. 
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BEST/1®—Design of a tool for 
computer system capacity planning 

by J. P. BUZEN, R. P. GOLDBERG, A. M. LANGER, E. LENTZ,* H. S. SCHWENK, 
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INTRODUCTION 

This paper presents the design philosophy used in the BEST/ 
1 computer performance evaluator. Essentially, BEST/1 is 
a modeling tool which has been developed to address the 
“what if” questions that arise when evaluating computer 
system performance. These questions are of importance in 
such areas as capacity planning, performance tuning, hard¬ 
ware vendor selection and new system design. 

From a design standpoint, the most significant features of 
BEST/1 are those which relate to ease of use. The following 
five objectives were considered most essential in this regard: 

• The analyst who is familiar with the basic terms and 
concepts of computer performance evaluation should 
require no special training in order to use BEST/1. That 
is, the internal algorithms and procedures used by 
BEST/1 should be insulated from the user to the max¬ 
imum extent possible. 

• The level of effort required to set up and evaluate a 
BEST/1 model should be commensurate with the ana¬ 
lyst’s objectives. First order models should be easy to 
set up and evaluate, with more refined models requiring 
progressively more effort in terms of data collection 
and analysis. 

• The input data required by BEST/1 should be easy to 
understand and obtain. Detailed measurements of in¬ 
ternal system activity should not be necessary when 
studying simple questions. 

• BEST/1 should operate interactively so that an analyst 
can have the benefit of working through a problem and 
exploring a number of related “what if’ questions dur¬ 
ing a single analysis session. The speed requirements 
associated with interactive operation are also important 
in sensitivity studies where a wide range of parameter 
settings have to be explored. 

• BEST/1 should generate a set of output reports which 
are easy to understand and apply. 


* Currently affiliated with Booz, Allen Hamilton, Inc., Los Angeles, Cali¬ 
fornia 

** Currently affiliated with IBM Corporation, San Jose, California 


In order to achieve these objectives, the BEST/1 design 
is based on an approach which is commonly employed in 
applications-oriented higher level languages. That is, a user 
visible front end has been developed which enables analysts 
to specify models and carry out analyses using concepts and 
terms which are generally familiar to individuals working in 
the area of computer performance evaluation. This user 
visible front end is then automatically translated into an 
internal BEST/1 analysis model which is insulated from the 
user in much the same way as machine language programs 
are insulated from the user of any higher level language. 
Thus, the most important design features of BEST/1 are 
those which relate to its user visible front end. These fea¬ 
tures are discussed in detail in the remainder of this paper. 

It should be noted that, in order to meet the objective of 
interactive operation, the internal BEST/1 analysis models 
are based on extensions to the theory of multiple class 
queueing network models. However, the BEST/1 user need 
not be concerned with the mathematical details of this the¬ 
ory, just as the user of an applications-oriented higher level 
language need not be concerned with the details of a partic¬ 
ular machine language. For this reason, the mathematical 
foundations of BEST/1 will not be emphasized in this paper. 
For discussions of the theory, see the references listed in 
the bibliography. 

BEST/1 MODELING CONCEPTS 

BEST/1 is an analysis tool which is capable of calculating 
the performance of almost any computer system on the basis 
of parameters which characterize that system’s hardware, 
software, and workload. The performance factors which 
are calculated by BEST/1 include throughput, response 
time, queue length, processor utilization, and so on. See 
Figure 1. 

The performance calculation function performed by 
BEST/1 is similar to the function performed by a benchmark 
experiment. That is, a BEST/1 analysis and a benchmark 
experiment both enable the analyst to evaluate the way a 
computer system will perform under a specific set of con¬ 
ditions. However, in the case of a benchmark experiment 
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Figure 1—Basic BEST/1 functionality 


this evaluation is made by running a set of programs on an 
actual system. In contrast, BEST/1 makes this evaluation 
on the basis of certain numerical parameters which are either 
measured directly or estimated by the analyst. 


COMPUTER 

SYSTEM 


Figure 2—The workload concept 


In the case of time sharing workloads, the analyst specifies 
the number of terminals and the average “think time” (i.e., 
the average time between the system’s response to a given 
transaction and the user’s initiation of the next transaction). 
The principal performance factors calculated by BEST/1 for 
these workloads are the response time per transaction and 
the throughput rate in transactions per hour. 

TRANSACTION PROCESSING WORKLOADS 


SYSTEM 

PERFORMANCE 


WORKLOADS 


THE WORKLOAD CONCEPT 

As illustrated in Figure 2, BEST/1 is based on the concept 
that performance issues arise when workloads of various 
types are processed by a computer system. In this context, 
the term “workload” refers to a stream of jobs or transac¬ 
tions arriving at a computer system from some external 
source. For example, a stream of batch jobs submitted by 
local or remote users could constitute a batch workload, a 
stream of transactions generated by time sharing terminals 
could constitute a time sharing workload, and so on. 

The significance of the workload concept lies in the fact 
that analysts are usually most concerned with performance 
factors which are defined on a “per workload” basis. This 
is because users normally perceive throughput and response 
time as being associated with individual workloads rather 
than with the system as a whole. In fact, it is generally 
meaningless to discuss throughput or response for an entire 
system in cases where several workloads are present. 

BEST/1 explicitly recognizes the central importance of 
workloads and uses the workload concept as a focal point 
for developing system models and organizing input param¬ 
eters. In particular, BEST/1 allows the analyst to specify a 
model in terms of certain generic workload types. Each 
workload type has an associated set of parameters which 
characterize the special features of that workload. The three 
principal workload types recognized by BEST/1 are dis¬ 
cussed below. 

TIME SHARING WORKLOADS 

The general nature of interactive or time sharing work¬ 
loads is illustrated in Figure 3. Essentially, each time sharing 
workload is associated with a set of terminals. The users at 
the terminals are assumed to generate transactions which 
arrive at the system and undergo processing. When a trans¬ 
action has been completed, a response is printed on the 
user’s terminal. The user then generates a new transaction 
and the cycle repeats. 


Transaction processing workloads are similar to interac¬ 
tive workloads in that the external load is assumed to be 
generated by users at terminals. However, instead of spec¬ 
ifying the external load in terms of number of terminals and 
think time, the analyst simply specifies a total arrival rate 
in “transactions per hour” as illustrated in Figure 4. Most 
analysts find it convenient to characterize transaction proc¬ 
essing workloads in this manner. 

Since the arrival rate is specified as an input parameter, 
and since throughput is equal to arrival rate (as long as 
system capacity is not exceeded), the most important per¬ 
formance value computed by BEST/1 for transaction proc¬ 
essing workloads is the response time. BEST/1 also calcu¬ 
lates the maximum attainable throughput rate in cases where 
this value is less than the arrival rate specified in the input 
file. 

BATCH PROCESSING WORKLOADS 

In the case of batch processing workloads, analysts are 
usually most interested in system performance during time 
periods when the input queues are backlogged (i.e., when 
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Figure 3—Time sharing workloads 
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Figure 4 —Transaction processing workloads 


However, the term “transaction” will be favored in most 
cases and should be understood to refer to either a batch 
job or an on-line transaction. 


SPECIAL WORKLOADS 


there is always at least one job waiting to be loaded into 
main memory). BEST/1 enables the analyst to specify batch 
workloads which have this “backlog” property. Schemati¬ 
cally, these workloads are represented as shown in Figure 
5. The closed loop in this figure indicates that, in a backlog 
situation, new jobs are loaded into main memory whenever 
an active job completes and memory space becomes avail¬ 
able. Both throughput and response time (turnaround time) 
are computed for these workloads. 

TOTAL NUMBER OF WORKLOADS 

The total number of workloads which may be included in 
a single BEST/1 model is not limited to three. For example, 
a single BEST/1 model could include two transaction proc¬ 
essing workloads, three batch processing workloads, and 
two time sharing workloads. Alternatively, a model could 
consist solely of one time sharing, one batch processing, or 
one transaction processing workload. In Release 4.0 the 
total number of workloads which may be included in a single 
BEST/1 model is equal to 9. 


JOBS AND TRANSACTIONS 

Each of the basic workload types discussed above makes 
use of the concept of a “job” or a “transaction”. Essen¬ 
tially, these terms both refer to a unit of work which arrives 
at a system from an external source, undergoes a certain 
amount of processing, and is eventually completed. 
Throughput is expressed in jobs or transactions per hour, 
and response time (or turnaround time) is expressed on a 
per job or per transaction basis. 

It is clear that the terms job and transaction refer to the 
same essential concept; however, the former term is more 
commonly used for batch processing workloads, whereas 
the latter is more common in the case of time sharing and 
transaction processing. In the following discussion of BEST/ 
1 , the two terms will be regarded as being interchangeable. 




SYSTEM 


NEW 

JOBS 


COMPLETED 

JOBS 


By suitable selection and adjustment of parameters, the 
three basic workload types defined above can be used to 
represent a number of special workloads which are of im¬ 
portance in evaluating particular systems. These workloads 
include periodic real time requests, system overhead activ¬ 
ities, and job streams with deadlines and critical paths. 


TREATMENT OF MULTIPROGRAMMING 

Figures 3-5 utilize a single box (SYSTEM) to represent 
the location of a transaction from the instant it arrives at a 
system to the instant it departs. The activity which takes 
place between these two points is described in greater detail 
in Figure 6. 

The interpretation of Figure 6 is straightforward. Small 
rectangles indicate the locations of queues, and circles rep¬ 
resent servers such as CPU’s or I/O devices. As indicated 
in the diagram, an arriving transaction may at first have to 
wait in the memory queue until space becomes available. 
Once it is loaded into memory, the transaction is placed in 
the CPU queue where it waits until a processor becomes 
available. The transaction then receives a burst of CPU 
processing which terminates when an I/O request is gener¬ 
ated. At this point, the transaction proceeds to the appro¬ 
priate I/O device queue. When it reaches the head of the 
queue associated with that device, an I/O transfer is initi¬ 
ated. Upon completing this transfer, the transaction returns 
to the CPU queue. The alternating cycle of CPU bursts and 
I/O transfers continues until the processing requirements of 
the transaction are satisfied. The transaction then terminates 
and leaves the system via the “COMPLETED TRANS¬ 
ACTIONS” arrow. 

BEST/1 enables the analyst to represent cases where sev¬ 
eral transactions are in main memory and active at the same 
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DEVICES 


Figure 5—Batch processing workloads 


Figure 6—Multiprogramming activity 
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time. One transaction may be using the CPU while another 
is using an I/O device. Similarly, several I/O devices may 
be providing service to several different transactions at the 
same time. 

This overlap of processing activity—which is generated 
by several transactions that are sharing the main memory of 
a single computer system—is generally referred to as mul¬ 
tiprogramming. Multiprogramming tends to raise the utili¬ 
zation of system resources and thereby improve certain 
measures of overall performance. However, the queueing 
delays associated with multiprogramming can have an ad¬ 
verse effect on other measures of system performance. 
BEST/1 has the ability to explicitly represent the multipro¬ 
gramming activity described above, and the analyst can use 
BEST/1 to determine the effectiveness of multiprogramming 
in various applications. 

THE SERVER CONCEPT 

As discussed in the preceding section, a job or transaction 
which has arrived at a system and has been loaded into main 
memory spends its time alternating between the following 
two states: 


drums which employ rotational position sensing (RPS). 
However, the number of actual data transfers which can 
take place at the same time is limited to the number of 
parallel I/O channels that are connected to a particular con¬ 
troller and string of devices. 

In cases such as these it is preferable to represent each 
disk and drum as a separate I/O server within BEST/1. That 
is, each disk and drum is assumed to be capable of inde¬ 
pendent simultaneous operation. The service time at each 
device must then be adjusted to account for the additional 
delays that arise because of channel contention. For exam¬ 
ple, disk service time—which is normally the sum of seek, 
rotational latency, and data transfer time—must be in¬ 
creased so that it becomes the sum of seek, rotational la¬ 
tency, data transfer, and channel queueing time. The in¬ 
creased service time represents a real effect since 
measurement devices would actually record the disk as 
being busy during channel queueing time. 

BEST/1 provides facilities for calculating the amount of 
channel queueing that will arise within a particular config¬ 
uration, and for automatically increasing I/O service times 
accordingly. Thus, channel contention is accounted for even 
though channels are not explicitly shown in Figure 6. 


• Using active processing resources such as CPU’s and 
I/O devices 

• Waiting to use such resources 

In the context of BEST/1, active processing resources are 
referred to as servers. That is, servers are points within a 
system where queues can form and where processing can 
be carried out. Each circle in Figure 6 corresponds to a 
different server. 

One of the most important aspects of Figure 6 is the 
assumption that each server in the network is capable of 
independent operation. That is, it is assumed that the CPU 
can be processing one request at the same time that an I/O 
server is processing another request. Likewise, it is assumed 
that two or more I/O servers can be processing requests at 
the same time. 

The assumption of simultaneous processing capability 
plays a critical role when the real I/O devices in a computer 
system are mapped into BEST/1 servers. For example, in 
the case of a string of high speed magnetic tape drives 
connected to a single controller and channel, it is only pos¬ 
sible for one device to be actively transferring data at any 
time. Hence the entire string of tape drives should be rep¬ 
resented as a single I/O server in Figure 6. On the other 
hand, if each tape drive were connected to a dedicated 
channel and controller, simultaneous operation would be 
possible and each device would be represented as a separate 
I/O server. 

The issues that arise when representing disks and drums 
are somewhat more complex. In virtually all modern disk 
drives and controllers, seek overlap is possible. This means 
that any number of disks can be carrying out independent 
seek operations at the same time. Simultaneous overlap of 
the rotational latency time is also possible for disks and 


THE DOMAIN CONCEPT 

As illustrated in Figure 7 the three principal components 
of a BEST/1 model are a set of workloads, a set of servers, 
and a set of memory domains. Essentially, memory domains 
are mechanisms for controlling the sharing of memory 
among various workloads. All multiple workload systems 
incorporate some mechanism of this type to prevent one or 
two workloads from monopolizing main memory in an un¬ 
controlled manner. For example, time sharing transactions 
may be assigned to a domain with a maximum multipro¬ 
gramming level of 5 while batch jobs are assigned to a 
domain with a maximum multiprogramming level of 7. This 
means that no more than 5 time sharing transactions and 7 
batch jobs would be allowed to enter main memory at a 



Figure 7—Components of a BEST/1 model 
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single time. Terms such as “partition”, “region” and “ini¬ 
tiator” are used in different operating systems to refer to 
the concepts of multiprogramming level and domain. 

The specific mechanisms used to implement the memory 
domain concept vary somewhat from one system to another. 
BEST/1 allows a wide range of mechanisms to be modeled. 
Essentially, the analyst can characterize domains in terms 
of their maximum multiprogramming levels, their average 
multiprogramming levels, or in terms of certain distribu¬ 
tions. 

Note that memory domains are closely associated with 
workloads. In particular, each workload must be assigned 
to some domain. BEST/1 permits several workloads to be 
assigned to the same domain; alternatively, each workload 
may be assigned to a separate domain if desired. 

As a final point, it should be clear from the preceding 
discussion that the memory queue shown in Figure 6 actually 
represents a set of queues, one corresponding to each do¬ 
main. BEST/1 treats the queues separately, and values such 
as queue length and waiting time are computed individually 
for each memory queue. 

BEST/1 INPUT PARAMETERS 

Figure 8 presents an example of all the input data needed 
by BEST/1 for a typical application. In this case, the system 
being evaluated is processing three workloads: Workload 1 
represents a stream of batch processing jobs, Workload 2 
represents a transaction processing application, and Work¬ 


load 3 represents the load generated by a set of time sharing 
users. 

The BEST/1 input parameters used to characterize this 
application are grouped together on a “per workload” basis. 
For each workload, there are a set of workload descriptors 
and a set of service requirements. These two sets of param¬ 
eters are discussed below. 


WORKLOAD DESCRIPTORS 

The workload descriptors in Figure 8 are used to specify 
the following information: 

• The workload type: batch processing (BP), time sharing 
(TS), or transaction processing (TP) 

• A label used to identify the workload on output reports 

• Information about the multiprogramming level of the 
domain which is associated with the workload 

• For time sharing and transaction processing workloads, 
information about the external factors which generate 
arriving transactions 

In all cases, the first workload descriptor in the formatted 
BEST/1 input file contains the workload type and the work¬ 
load label. The second descriptor characterizes the multi¬ 
programming level of the workload’s domain. For Workload 
1 , the average multiprogramming level attained during times 
when there is a backlog in the input queue is specified as 


•WORKLOAD 1- 
BP 

3.8 

-WORKLOAD 2- 
TP 
4.0 
4000.0 

WORKLOAD 3 
TS 
6.0 
30.0 
25.0 


-DESCRIPTORS- 

BATCH PROCESSING 
AVERAGE MPL 

-DESCRIPTORS. 

DATA BASE TRANS 
MAXIMUM MPL 
TRANSACTIONS/HR 

-DESCRIPTORS- 

TIME SHARING USERS 
MAXIMUM MPL 
NO. OF TERMINALS 
THINK TIME (SECS) 


SERVER 

WORKLOAD 1 

WORKLOAD 2 

WORKLOAD 3 

1 CPU—370/168 

13000.0 

205.0 

320.0 

2 DRUM 1 

1099.0 

83.0 

92.0 

3 DRUM 2 

1940.0 

147.0 

175.0 

4 TAPE 

395.0 

0.0 

0.0 

5 TAPE 

496.0 

0.0 

0.0 

6 2314 

747.0 

0.0 

0.0 

7 2314 

204.0 

0.0 

0.0 

8 3330 SYSTEM PACK 

4642.0 

13.0 

80.0 

9 3330/ASPQ 

12461.0 

0.0 

0.0 

10 3330/ASPQ 

12462.0 

0.0 

0.0 

11 3330 SCRATCH 

2474.0 

64.0 

42.0 

12 3330 SCRATCH 

2474.0 

62.0 

46.0 

13 3330 SWAP AND USER 

3034.0 

150.0 

100.0 

14 3330 SWAP AND USER 

3035.0 

150.0 

100.0 

15 3330 MODI USER PACK 

1073.0 

27.0 

200.0 

16 3330 MODI USER PACK 

1073.0 

29.0 

0.0 

17 3330 MOD 11 USER PACK 

1727.0 

49.0 

0.0 

18 3330 MOD 11 USER PACK 

1727.0 

52.0 

0.0 


Figure 8—Basic input file 
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3.8. Workloads 2 and 3 are assigned maximum multipro¬ 
gramming levels of 4 and 6 respectively. 

Additional descriptors are used in Workloads 2 and 3 to 
characterize the external factors which generate arriving 
transactions. For Workload 2, these factors are character¬ 
ized by specifying an average arrival rate of 4000 transac¬ 
tions per hour. For Workload 3, the external factors are 
specified as 30 terminals with average think times of 25 
seconds. 

SERVICE REQUIREMENTS 

The second set of input parameters that are provided for 
each workload are referred to as the workload’s service 
requirements. Essentially, these parameters specify the total 
amount of service time that is required, per transaction, at 
each server in the system. For example, each Workload 1 
transaction (i.e., each batch job) requires an average of 
13000 msec of processing at Server 1 (the CPU), 1099 msec 


of processing at Server 2 (Drum 1), 1940 msec of processing 
at Server 3 (Drum 2), and so on. The service requirements 
per transaction are specified separately for each workload 
as indicated in Figure 8. 

Note that the service requirements in Figure 8 refer only 
to the time that a transaction spends actually using a server. 
The time spent waiting in a queue before reaching the server 
is not included in the input data, but is instead computed by 
BEST/1 and used to generate output reports. 

EXTENDED EXAMPLE 

BEST/1 enables the analyst to extend the basic system 
description of Figure 8 in a number of ways. These exten¬ 
sions are introduced by associating additional descriptors 
with each workload, and also by adding certain “special 
server parameters” to the end of the input file. Figure 9 
provides an indication of the range of extensions which are 
available to the analyst. 


-WORKLOAD 1.DESCRIPTORS- - - 

BP BATCH PROCESSING 
3.8 AVERAGE MPL 
3.0 PRIORITY 

■WORKLOAD 2.DESCRIPTORS- - - 


TP 

DATA BASE INQUIRY 

3200.0 

TRANSACTIONS/HR 

1.0 

PRIORITY 

1.0 

DOMAIN NUMBER 

•WORKLOAD 3- 

-DESCRIPTORS- 

TS 

TIME SHARING USERS 

6.0 

MAXIMUM MPL 

30.0 

NO. OF TERMINALS 

25.0 

THINK TIME (SECS) 

2.0 

PRIORITY 

■WORKLOAD 4- 

-DESCRIPTORS- 

TP 

DATA BASE UPDATE 

800.0 

TRANSACTIONS/HR 

1.0 

PRIORITY 

1.0 

DOMAIN NUMBER 


SERVER 

WORKLOAD 1 

WORKLOAD 2 

WORKLOAD 3 

WORKLOAD 4 

1 CPU—370/168 

13000.0 

150.0 

320.0 

425.0 

2 DRUM 1 

1099.0 

75.0 

92.0 

115.0 

3 DRUM 2 

1940.0 

125.0 

175.0 

235.0 

4 TAPE 

395.0 

0.0 

0.0 

0.0 

5 TAPE 

496.0 

0.0 

0.0 

0.0 

6 2314 

747.0 

0.0 

0.0 

0.0 

7 2314 

204.0 

0.0 

0.0 

0.0 

8 3330 SYSTEM PACK 

4642.0 

13.0 

80.0 

13.0 

9 3330/ASPQ 

12461.0 

0.0 

0.0 

0.0 

10 3330/ASPQ 

12462.0 

0.0 

0.0 

0.0 

11 3330 SCRATCH 

2474.0 

20.0 

42.0 

240.0 

12 3330 SCRATCH 

2474.0 

20.0 

46.0 

230.0 

13 3330 SWAP AND USER 

3034.0 

100.0 

100.0 

350.0 

14 3330 SWAP AND USER 

3035.0 

100.0 

100.0 

350.0 

15 3330 MODI USER PACK 

1073.0 

15.0 

200.0 

75.0 

16 3330 MODI USER PACK 

1073.0 

15.0 

0.0 

85.0 

17 3330 MOD11 USER PACK 

1727.0 

20.0 

0.0 

165.0 

18 3330 MOD11 USER PACK 

1727.0 

21.0 

0.0 

176.0 


DOMAIN 1.DESCRIPTORS- - 

LABEL DATA BASE TRANS 
4.0 MAXIMUM MPL 


■SERVER 1.DESCRIPTORS. 

LABEL CPU—370/168 

2.0 MULTIPLE SERVER 

Figure 9—Extended input file 






















BEST/1 


453 


WORKLOAD DIFFERENTIATION 

The most obvious difference between Figure 8 and Figure 
9 is the fact that a fourth workload has been added. In this 
particular case, the additional workload is not intended to 
represent a new application being added to the system, al¬ 
though additional workloads could obviously be used for 
such purposes. Rather the new workload represents a more 
detailed model of the same system represented in Figure 8. 

In the case of Figure 8 the transaction processing appli¬ 
cation is represented, using Workload 2, as a single input 
stream arriving at a rate of 4000 transactions per hour. 
BEST/1 will treat this workload as a single entity and will 
report response time, queue length, and other performance 
values for the workload as a whole. 

Suppose, however, that the analyst now wishes to explic¬ 
itly represent the fact that 80 percent of the Workload 2 
input stream corresponds to data base inquiry transactions, 
and 20 percent corresponds to data base update transactions. 
In other words, suppose the analyst wishes to obtain sepa¬ 
rate performance values for each of the two transaction 
types. 

In this case the analyst can use two separate workloads 
to represent the transaction processing application. As illus¬ 
trated in Figure 9, Workload 2 is now used to represent the 
data base inquiry transactions. The arrival rate is set to 3200 
transactions per hour (80 percent of 4000), and the service 
requirements are explicitly shown. In addition, Workload 4 
is now used to represent the data base update transactions. 
The arrival rate is 800 transactions, and a different set of 
service requirements are provided. BEST/1 will then be able 
to compute response time, throughput, and other perform¬ 
ance values for each transaction type. 

LEVELS OF DETAIL 

It should be emphasized at this point that the model rep¬ 
resented by Figure 9 is not intrinsically “more correct” than 
the model represented by Figure 8. Both are “correct” in 
the sense that they both enable the analyst to investigate a 
certain set of questions concerning the performance of a real 
system. Since the model associated with Figure 9 is more 
detailed, a more detailed set of questions can be studied. 
However, the analyst is required to provide a more detailed 
set of input data in this case: specifically, the separate ser¬ 
vice requirements for each of the two transaction types. 

Because of the ease with which the level of detail can be 
altered, BEST/1 enables the analyst to select an initial level 
of detail, run through a number of cases, and then increase 
the level of detail if necessary for those cases which warrant 
further analysis. 

The ability to adjust the level of detail in a BEST/1 anal¬ 
ysis is also important in applications involving capacity plan¬ 
ning, design of new systems, and other situations where 
performance prediction is involved. In these cases the in¬ 
formation available to the analyst tends to become more and 
more detailed as time progresses. BEST/1 thus enables the 
analyst to use the most detailed model that is feasible at 


each stage, and to increase the detail in the model as more 
refined information becomes available. 


SHARED DOMAINS 

Another important extension of Figure 8 which is illus¬ 
trated in Figure 9 is the concept of the shared domain. It is 
implicitly assumed that each workload in Figure 8 has a 
separate domain and a separate memory queue. However, 
this is not the case for Workloads 2 and 4 in Figure 9. 
Rather, these two workloads are assumed to compete for 
the single domain which was assigned to Workload 2 in 
Figure 8. 

The concept of two workloads sharing a single domain is 
easy to represent in BEST/1. As illustrated in Figure 9, this 
is done by placing a domain identifier rather than a multi¬ 
programming level in the descriptors of Workloads 2 and 4. 
A separate domain descriptor is then added to the end of 
the input file to indicate the multiprogramming level asso¬ 
ciated with that shared domain. This is shown in Figure 9. 

Note that the analyst is not required to use a shared 
domain in this case. For example, Workload 2 could have 
been assigned a maximum multiprogramming level of 3, and 
Workload 4 could have been assigned a maximum multipro¬ 
gramming level of 1. This corresponds to a situation where 
each workload has a separate domain. BEST/1 will, of 
course, calculate a different set of performance values in 
this case. The analyst can then determine which memory 
allocation strategy is most appropriate in terms of the per¬ 
formance objectives of the original system. 

PRIORITIES 

BEST/1 permits the analyst to associate a CPU dispatch¬ 
ing priority with each workload. In the case of Figure 8 no 
priorities were specified so BEST/1 assigned the same de¬ 
fault priority to all workloads. Figure 9 illustrates the way 
priorities are added to the workload descriptors. As indi¬ 
cated in the figure, the transaction processing workloads 
have highest priority, the time sharing workload has inter¬ 
mediate priority, and the batch processing workload has 
lowest priority. 

In cases where priorities are specified, BEST/1 uses the 
conventional “preemptive-resume” dispatching algorithm 
found in most operating systems. In other words, a high 
priority request which arrives at the CPU interrupts and 
preempts any lower priority requests that may be present. 
Lower priority requests resume processing from the point 
of preemption after higher priority requests release the CPU. 

MULTIPROCESSORS 

BEST/1 input files sometimes include “special server pa¬ 
rameters” which refer to the servers within the system 
rather than individual workloads. The number of CPU’s in 
a multiprocessor configuration is such a parameter. Figure 
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Principal Results Report 

CPU and I/O Utilization Report 

Response Time Profile 

Average Queue Length Report 

Average Number Waiting Report 

Waiting Time Profile 

Service Rate Degradation Report 

Server Residency Profile 

Queue Length Distribution Report 

Concurrency Report 

Memory Report 

Throughput Report 

Server Report 

Waiting Time Percentages Report 


Figure 10—BEST/1 output reports 


9 illustrates a case where two CPU’s are present. Special 
server parameters are used to represent other system fea¬ 
tures such as servers whose service rate varies as an arbi¬ 
trary function of load. 


RESPONSE TIME PROFILE 

Figure 12 illustrates the Response Time Profile report 
generated by BEST/1 from the input data shown in Figure 
8 . Essentially, this report gives the analyst a breakdown of 
the response time values that appear in the Principal Results 
report. That is, this report gives the response time, in mil¬ 
liseconds, at each server in the system. By adding up these 
individual response times, the total response time per trans¬ 
action is obtained. 

Note that main memory is represented in this report by 
“Server O”. Thus, the response time at Server O corre¬ 
sponds to the delay a transaction experiences before being 
loaded into its memory domain. This delay only arises when 
the number of transactions at the system (for a given work¬ 
load) exceeds the maximum multiprogramming level of the 
domain associated with that workload. 

The Response Time Profile is particularly useful when it 
is necessary to determine why a workload’s response time 
value is so high, and how important each server and queue 
really is insofar as overall response time is concerned. 


GENERATION OF OUTPUT REPORTS 

Once a particular set of input data has been specified, the 
BEST/1 user may request an evaluation. This will cause 
BEST/1 to analyze the data and generate the “Principal 
Results” report. After examining this report, the analyst 
may request one or more additional reports. 

BEST/1 generates a number of different reports which 
provide detailed information about the performance of the 
system being evaluated. The report titles are listed in Figure 
10 . 


INTERACTIVE DIALOG 

The primary mode of operation for BEST/1 is through 
interactive time sharing terminals. When operating in this 
mode, the analyst has access to a variety of functions. These 
functions are requested through an interactive dialog that 
has been extensively refined and enhanced during several 
years of actual use. 

The complete interactive dialog is specified in the BEST/ 
1 User’s Guide. Some typical functions which are easy to 
carry out using the dialog are listed below: 


PRINCIPAL RESULTS 

Figure 11 illustrates the Principal Results report generated 
by BEST/1 from the input data shown in Figure 8. This 
report is organized by workload. For each workload, the 
report indicates the average response time per transactions, 
the average throughput in transactions per hour, and the 
CPU utilization associated with that workload. Total CPU 
utilization is also reported. 

For example, in Figure 11 the batch processing throughput 
is 102 jobs per hour and the response times for transaction 
processing and time sharing are 3.17 seconds and 3.31 sec¬ 
onds respectively. 


• Add servers or workloads 

• Change server speed 

• Change arrival rate, number of terminals, CPU dis¬ 
patching priority, multiprogramming level, etc. 

• Request one or more optional reports 

• Save an input file for future use 

• Direct output to any desired device 

• Execute a pre-defined set of interactive commands 
stored in a file 

It should be emphasized that this is only a partial list of the 
functionality available to the BEST/1 user. 

In all cases, the interactive dialog has been structured 
using concepts that can be easily and directly understood by 


♦♦♦PRINCIPAL RESULTS*** 

WORKLOAD 

RESPONSE TIME 

THROUGHPUT 

%CPU 

1 BATCH PROCESSING 

133.98 SEC 

102. PER HOUR 

36.9% 

2 DATA BASE TRANS 

3.17 SEC 

4000. PER HOUR 

22.8% 

3 TIME SHARING USERS 

3.31 SEC 

3815. PER HOUR 

33.9% 



TOTAL CPU UTILIZATION 

= 93.6% 


Figure 11—Principal results report 
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RESPONSE TIME PROFILE BY 



WORKLOAD 


SERVER 

1 

2 

3 

0 MEMORY 

0.0 

838.1 

221.5 

1 CPU—370/168 

65680.5 

1159.7 

1873.0 

2 DRUM 1 

1410.7 

107.0 

119.4 

3 DRUM 2 

3244.4 

242.0 

292.4 

4 TAPE 

397.4 

0.0 

0.0 

5 TAPE 

500.2 

0.0 

0.0 

6 2314 

757.8 

0.0 

0.0 

7 2314 

204.3 

0.0 

0.0 

8 3330 SYSTEM PACK 

5865.2 

17.1 

105.6 

9 3330/ASPQ 

17053.1 

0.0 

0.0 

10 3330/ASPQ 

17054.9 

0.0 

0.0 

11 3330 SCRATCH 

3005.0 

79.5 

52.3 

12 3330 SCRATCH 

3012.5 

77.2 

57.4 

13 3330 SWAP AND USER 

4675.4 

231.8 

156.4 

14 3330 SWAP AND USER 

4677.1 

231.8 

156.4 

15 3330 MODI USER PACK 

1477.6 

37.3 

277.6 

16 3330 MODI USER PACK 

1137.1 

31.4 

0.0 

17 3330 MOD11 USER PACK 

1909.1 

55.4 

0.0 

18 3330 MOD11 USER PACK 

1916.5 

59.0 

0.0 

TOTAL RESPONSE TIME 

133978.6 

3167.1 

3312.1 


Figure 12—Response time profile 


the computer performance analyst. Knowledge of the un¬ 
derlying mathematical theory is not required to set up BEST/ 
1 models or exercise the interactive dialog. 


SUBROUTINE INTERFACE 

Users are sometimes interested in carrying out complex 
iterative analysis procedures which repeatedly invoke 
BEST/1. For example, a user may wish to increase a work¬ 
load’s arrival rate until the resulting response time reaches 
some threshold. 

Some cases can, of course, be treated using the BEST/1 
interactive dialog. However, it is sometimes preferable to 
simply write a program which repeatedly calls BEST/1, ana¬ 
lyzes the output, and adjusts certain BEST/1 input param¬ 
eters until a condition has been satisfied or a set of cases 
has been examined. 

In order to facilitate this process, there is a special inter¬ 
face which allows BEST/1 to be called as a subroutine by 
a FORTRAN program. The FORTRAN program can then 
be executed interactively or in batch processing mode. 
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Job scripts—A workload description 
based on system event data* 

by ROBERT L. MEAD and HERBERT D. SCHWETMAN 

Purdue University 
West Lafayette, Indiana 


INTRODUCTION 

Job scripts, a detailed description of a job’s resource de¬ 
mands, can serve not only as input for a simulation model, 
but as a representation of the system’s workload as well. 
Heretofore, most representation efforts have dealt with 
either benchmark or synthetic job techniques. This paper 
presents a workload representation based on a set of job 
scripts, each representing a single job in the workload, that 
is designed for use with simulation models. This represen¬ 
tation differs from the synthetic job approach in the form of 
the representation, the intended usage, and the level of de¬ 
tailed involved. The scripts have been used as input for a 
trace-driven simulation model of the CDC6500 system at 
Purdue University and have also been used to increase the 
understanding of the execution time characteristics of pro¬ 
grams. 

The accurate representation of the workload is an impor¬ 
tant problem in the modeling and evaluation of computer 
systems. In 1972, Grenander and Tsao 1 stated, “We believe 
that no real significant advance in the evaluation of systems 
can be expected until some breakthrough is made in the 
characterization of the workload.” Since that time, ad¬ 
vances have been made in understanding program behavior 
in terms of memory reference behavior, 2-6 and some work 
has appeared on CPU usage behavior. 7 Job scripts can be 
used to increase the understanding of how programs utilize 
CPUs, data channels, and I/O processors. One outgrowth of 
this investigation has been activity plots which depict a 
program’s file and device usage patterns, in much the same 
way that page reference maps show the pattern of memory 
demands. 

The paper begins by summarizing earlier efforts in work¬ 
load representation. The form of the job script and the gen¬ 
eration techniques are presented. The issues of accuracy 
and the reproducibility of the job script are discussed, and 
data is presented to demonstrate that the scripts are repro¬ 
ducible and accurate. The advantages, limitations, and use¬ 
fulness of job scripts are presented, along with some ex¬ 
amples of the activity plots mentioned previously. 


* This work was supported in part by NSF grant GJ-41289. 


PREVIOUS WORK 

As early as 1965 it was apparent that carefully choosing 
a representative workload was important when evaluating 
a computer system. Joslin 8 ’ 9 stressed the importance of 
knowing the nature of a workload to be represented by a 
benchmark and demonstrated that the performance of a 
system could vary depending upon the benchmark chosen. 
DeMeis and Weizer 10 represented interactive programs with 
pseudo-programs that were built around a loop of clocking 
instructions. Also in 1969, Buchholz 11 first introduced the 
idea of a synthetic program, a parameterized cyclic program 
that performed no useful work, but that was designed to 
consume a desired amount of CPU and I/O time. Wood and 
Forman 12 used accounting data and the output from a hard¬ 
ware monitor to parameterize synthetic programs in order 
to test the throughput of IBM 360 configurations. Synthetic 
programs have also been used in a number of other efforts. 
Schwetman and Browne 13 used an event driven software 
monitor to provide parameters for a synthetic job stream 
generator, and Kemighan and Hamilton 14 successfully used 
synthetic jobs to represent individual jobs in a standard 
benchmark. Sreenivasan and Kleinman 15 describe a tech¬ 
nique for generating a synthetic workload by matching the 
joint probability densities of the real and synthetic work¬ 
loads. Agrawala and Mohr 16 present a general model involv¬ 
ing mixture distributions in a chosen n-space, leading to a 
representation that can be validated statistically. Cohen 17 
described a job catalog that can be used to describe a 
benchmark set of programs. 

The concept of workload representation in the form of 
trace data for trace-driven simulation models was proposed 
by Cheng 18 in 1969. Sherman 19 and Sherman et al. 20 suc¬ 
cessfully used a trace-driven model to investigate deadlock 
algorithms and CPU scheduling strategies. A forerunner to 
the job script was presented by Keim and Schwetman, 21 and 
was used to produce statistical descriptions of CPU and 
system routine usage intervals. 

THE JOB SCRIPT AS A WORKLOAD 

REPRESENTATION 

The basic concept underlying the job script is that of a 
CPU — I/O cycle. As a job executes, it typically uses the 
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CPU for a period of time and then requests an I/O operation. 
At the completion of the I/O operation, or immediately after 
the issuance of the I/O request in the case of overlapped 
I/O activity, the CPU is again used until a subsequent 
I/O request. This pattern of behavior continues until the job 
completes. 

A job script represents this behavior in terms of an ex¬ 
tended sequence of alternating CPU intervals and I/O se¬ 
quences. CPU intervals represent the total amount of CPU 
time actually consumed between successive I/O requests 
and the I/O sequences consist of the resource usage intervals 
involved in I/O operations: I/O processor intervals and data 
transfer intervals. The total resource usage of a job is thus 
represented by CPU, I/O processor, and data channel usage 
intervals. It is important to note that we are not dealing with 
a virtual memory system. The job script representation as¬ 
sumes that all of the memory a job will need at a given time 
is allocated to the job in a contiguous block. It is possible 
however, for an executing job to request additional memory, 
or to release memory (at the end of the block) that is no 
longer needed. 

Additional detail concerning the execution time charac¬ 
teristics of jobs is also included by the insertion of infor¬ 
mation entries in the script. These entries provide informa¬ 
tion about the type of I/O operation requested and the 
associated file name, the number of characters transferred 
during an I/O operation, the name of the I/O processor 
routine used, the current amount of memory being held by 
a job, and the name of the current job step (control card). 
This combination of resource usage intervals and informa¬ 
tion entries provides a complete and detailed description of 
the demands placed on the system by an individual job. 

Ideally, the job scripts form a load independent represen¬ 
tation, since the time a job spends waiting for resources is 
eliminated. In practice however, the processing load does 
affect the length of some resource usage intervals to a small 
degree, due to such things as varying seek times and rota¬ 
tional delays for disk accesses, and varying amounts of 
system overhead. The effect of the load on the script, along 
with the issues of accuracy, reproducibility, and usefulness 
of job scripts are discussed in subsequent sections of this 
paper. 

The job script approach differs from the synthetic job 
approach on one key point. The script technique is essen¬ 
tially a bottom-up approach, where every system event of 
interest is examined in order to create the scripts. This 
differs from the formation of synthetic jobs where informa¬ 
tion with little detail (e.g., accounting data) is used as a 
starting point, and the detail required to make the job real¬ 
istic is introduced. It is important to note that the data from 
which the scripts are generated is gathered at the level at 
which a job interacts with the system. This implies that, as 
long as we restrict ourselves to monitoring techniques that 
are external to the job itself, we cannot increase the level of 
detail included in the representation. 

JOB SCRIPT GENERATION 

The generation of the job scripts involves the use of a 
software monitor and two analysis programs. A software 


system monitor first collects event data while the system is 
in operation. The event data is processed to produce a tem¬ 
porary file consisting of all requests, assignments, and re¬ 
leases of system resources for each job, plus all of the 
additional information to be included in the job script. The 
temporary file is processed to eliminate all queueing delays 
experienced by each job and to remove any activity solely 
concerned with load dependent activities. In the system 
being studied, it is possible for a job to be preempted from 
memory prior to completion and then to be returned to 
memory at a later time. Since we are not dealing with a 
virtual memory system, when a job is to be preempted from 
memory, it is necessary to save the contents of the job’s 
entire memory allocation, plus the contents of all hardware 
registers associated with the job. Thus both the preemption 
and subsequent resumption require I/O processors and data 
channels to copy the contents of memory to and from a 
disk. The number of preemptions incurred by a job depends 
on the system load, and as such they are not included in the 
job script. These preemptions and resumptions represent 
the majority of the load dependent activity eliminated from 
the job scripts. 

It should be noted that the removal of this load dependent 
behavior does not imply that this activity is ignored in the 
simulation model driven by the scripts. The activity asso¬ 
ciated with job preemption and resumption is included in 
the trace-driven simulation model but is generated from 
probability distributions when the simulation model decides 
a job preemption or resumption should take place. The elim¬ 
ination of this load dependent behavior from the script rep¬ 
resentation thus allows for greater flexibility in the simula¬ 
tion model. 

The details of the software monitor can be found in Ref¬ 
erences 22 and 23, and a modified version of the data ana¬ 
lyzer described in References 23 and 24 is used to generate 
the temporary file. The basic problem involved in the re¬ 
duction of the data on the temporary file is the recognition 
of the resource usage intervals. This problem is complicated 
by the presence of overlap between CPU and I/O activity 
and by the fact that several distinct I/O sequences may be 
in progress simultaneously. As will be seen in the example 
below, this implies that the order of the intervals in the job 
script is not necessarily the order of occurrence during ex- 


Event 




Event 




Number 

Time 

Event 

Resource 

Number 

Time 

Event 

Resource 

1 

0 

Req 

CPU 

16 

32 

Req 

CHOI (3) 

2 

1 

Ass 

CPU 

17 

40 

Ass 

CHOI (3) 

3 

4 

Re 1 

CPU 

18 

43 

Req 

CPU 

4 

4 

Req 

CPU 

19 

43 

Ass 

CPU 

5 

10 

Ass 

CPU 

20 

46 

Re 1 

CPU 

6 

13 

Req 

I OP 

21 

55 

Re 1 

CHOI (3) 

7 

15 

Ass 

IOP 5 

22 

56 

Req 

CPU 

8 

17 

Re 1 

CPU 

23 

58 

Ass 

CPU 

9 

19 

Req 

CH00 (5) 1 

24 

59 

Re 1 

IOP 3 

10 

20 

Req 

CPU 

25 

61 

Re 1 

CPU 

11 

22 

Ass 

CPU 

26 

63 

Re 1 

CH00 (5) 

12 

25 

Ass 

CH00 (5) 

27 

64 

Re 1 

IOP 5 

13 

27 

Req 

IOP 

28 

67 

Req 

CPU 

14 

28 

Ass 

IOP 3 

29 

67 

Ass 

CPU 

15 

29 

Re 1 

CPU 

30 

70 

Req 

IOP 

1 I/O processor number in parenthesis 


Figure 1—Example of event records in the intermediate file 
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Script 


Interva1 

Interva1 

Bounds 1 

Entry 

Resource 

Length 

Start 

End 

1 

CPU 

6 

2 

6 

2 

IOP 5 

4 

7 

9 

3 

CHOO 

38 

12 

26 

4 

IOP 5 

1 

26 

27 

5 

CPU 

9 

6 

13 

6 

IOP 3 

4 

14 

16 

7 

CHOI 

15 

17 

21 

8 

IOP 3 

4 

21 

24 

9 

CPU 

11 

13 

30 

1 Start and 

End denote event records 

in 

Figure 

1 that define the interval 


Figure 2—Example of a job script 


ecution. Figure 1 contains an example of the event records 
that might appear in the temporary file for a portion of a job. 
The job script entries subsequently generated appear in Fig¬ 
ure 2. Note that the intervals corresponding to script entries 
5, 6, 7, and 8 all occur before the interval corresponding to 
entry 4. The order in the script does, however, correspond 
to the sequence that would result if overlap of I/O processors 
did not take place. 

ACCURACY AND REPRODUCIBILITY OF JOB 

SCRIPTS 

In order to be of benefit, the scripts must be shown to be 
both an accurate and reproducible representation of a job’s 
behavior. The correctness of the script was determined by 
listing the event records from which the scripts are pro¬ 
duced and verifying that the resultant script accurately in¬ 
terprets the event records. Reproducibility is more difficult 
to verify, mainly because of variation in resource usage that 
is not under the control of the script generation process. 
Two major reasons for variations of this nature are varying 
seek times and rotational delays in data channel intervals 
and varying amounts of system overhead included in CPU 
usage. 

As the monitor is unable to measure the seek time for 
disk accesses, the placement of files on the system disks 
and the load on the system can cause access times to vary 
greatly for different executions of the same job. This leads 
to fluctuations in the scripts. Rotational delay times also 
vary, but over time (assuming we are observing a lengthy 


job) these variations should tend to average out. In addition, 
the monitor does not record all CPU changes to supervisor 
mode, so the length of recorded CPU intervals can vary 
depending on the amount of system overhead occurring 
while a job is using the CPU. 

To ascertain the degree of variation in the job scripts, four 
jobs of differing characteristics were each executed four 
times in a monoprogrammed environment. Scripts were gen¬ 
erated for each of these jobs and analyzed using a statistics 
gathering program. For each job script the following quan¬ 
tities were determined: 

(1) memory residency time 

(2) adjusted memory residency time (memory residency 
t time minus the time during which only load dependent 

activity occurred) 

(3) total work of the job (CPU, I/O processor, and data 
channel time) 

(4) total CPU time 

(5) total I/O processor time 

(6) total data channel time 

(7) total number of I/O transfer units (1 transfer unit = 
640 characters) 

(8) total number of I/O sequences 

Table I contains the statistics compiled for one copy of each 
of the four jobs. A maximum relative error for each quantity 
for each job was determined by using the smallest observed 
value of each quantity as the correct value. As can be seen 
from the summary of results in Table II, all Of the measured 
quantities are reasonably consistent, with the exception of 
the data channel times. This is not surprising, however, due 
to the problems of seek times and rotational delay discussed 
above. We have attempted to determine the reason for the 
rather high degree of variation in the data channel time for 
job 2 (20.6%), but the difference seems to be caused only 
by the inherent variation within the system. 

It is not usually possible to execute jobs in a monopro¬ 
grammed environment, however, so the script must be able 
to eliminate variation caused by the load in a multipro- 
grammed environment. To determine the effects of multi¬ 
programming, the same four jobs were executed four times 
in a multiprogrammed environment. Job scripts were again 
generated, and the same set of statistics collected. The max¬ 
imum relative errors determined for the multiprogrammed 
jobs appear in Table III. Once again most of the measure¬ 
ments are consistent. Memory residency times fluctuate to 
a greater extent, but the fact that the total work of each job 


TABLE I.—Sample Statistics of Monoprogrammed Jobs 



Job 1 

Job 2 

Job 3 

Job 4 

Memory Residency Time 1 

8. 321 

36.360 

52.169 

14.864 

Adj. Memory Res. Time 1 

7.752 

35.099 

51.353 

14.298 

Total Work 1 

8.008 

35.895 

62.470 

15.867 

CPU Time 1 

1.810 

18.899 

26.060 

9.447 

I/O Processor Time 1 

1.338 

1.552 

3.027 

0.873 

Data Channel Time 1 

4.860 

15.444 

33.383 

5.547 

No. I/O Sequences 

183 

268 

762 

151 

Total Transfer Counts 2 

814 

1461 

2358 

1115 

1 seconds 





| 2 1 transfer count = 640 characters transferred 



TABLE II—Maximum Relative Errors over Four Runs 
Monoprogrammed Jobs 



Job 1 

Job 2 

Job 3 

Job 4 

Memory Residency Time 

5.02 

7.62 

3.82 

3.92 

Adj. Memory Res. Time 

5.2 

8.3 

3.7 

3.8 

Total Work 

6.0 

8.2 

2.3 

4.7 

CPU Time 

1.5 

0.2 

0.7 

0.3 

I/O Processor Time 

1.3 

2. 1 

2.8 

1 . 1 

Data Channel Time 

9.7 

20.6 

4.1 

12.6 

No. I/O Sequences 

0.0 

0.7 

2. 1 

1.3 

Total Transfer Counts 

0.0 

0.0 

0.1 

0 . 1 



















460 


National Computer Conference, 1978 


TABLE III—Maximum Relative Errors over Four Runs TABLE V—Comparison of Totals 

Monoprogrammed Jobs Multiprogramming over Monoprogramming 



Mono- 

programmed 

Multi¬ 

programmed 

Increase 

Memory Residency Time 

449.371 

577.242 

28.5% 

Adj. Memory Res. Time 

436.078 

535.144 

22.7 

Total Work 

490.942 

516.476 

5.2 

CPU Time 

225.562 

232.270 

3.0 

I/O Processor Time 

27.092 

28.824 

6.4 

Data Channel Time 

238.468 

253.344 

6.2 

No. I/O Sequences 

5440 

5431 

-0.2 

Total Transfer Counts 

22,991 

22,999 

0.0 



Job 1 

Job 2 

Job 3 

Job 4 

Memory Residency Time 

22. 1% 

7.3% 


31.7% 

Adj. Memory Res. Time 

16.0 

9.9 


37. 4 

Total Work 

6.7 

2.8 


10.6 

CPU Time 

6.4 

1. 1 


3.3 

I/O Processor Time 

10.7 

6.7 


18.9 

Data Channel Time 

8.6 

7.5 

10.8 

20.0 

No. I/O Sequences 

4.4 

1.5 


1.4 

Total Transfer Counts 

0.4 

0.0 


0.0 


exhibits a smaller variation (relative to the memory resi¬ 
dency times) indicates that a majority of the effects of mul¬ 
tiprogramming have been eliminated. 

Finally, the job scripts generated in the monoprogrammed 
and multiprogrammed environments were compared. The 
average value of each measure for each job was determined 
in both environments, and the percentage increase caused 
by the multiprogramming, relative to the monoprogrammed 
value, was computed (i.e., (multi-mono)/mono). The results 
of these comparisons appear in Table IV. In addition, the 
sum of each quantity over all 16 jobs was determined for 
each environment, and the increase in the multiprogrammed 
environment determined. These results appear in Table V. 
It seems reasonable to expect a slight increase in observed 
resource usage due to increased overhead and seek times in 
a multiprogrammed environment, and the data confirms this 
expectation. 

Based on the data in Tables II-V, the job scripts appear 
to be reproducible and relatively insensitive to the load on 
the system. Those variations that are present seem to be 
inherent to the system itself, and as such, any workload 
characterization effort would be unable to prevent such var¬ 
iations from being included in the workload description. 

ADVANTAGES AND USES OF THE JOB SCRIPT 

The primary advantage of the job script as a workload 
representation lies with its completeness. The scripts con¬ 
tain a wealth of detail, and they can be analyzed in a number 
of different ways. File usage patterns can be studied, in¬ 
cluding the types of operations and the number of characters 
transferred in each operation. The characteristics of each 
job step can be investigated by using the job step name 
entries. Since each resource usage interval is included, the 


TABLE IV.—Average Percentage Increase Multiprogrammed over 
Monoprogrammed 



Job 1 

Job 2 

Job 3 

Job 4 

Memory Residency Time 

64.2% 

42.8% 

4.5% 

58.0% 

Adj. Memory Res. Time 

50.5 

35.8 

3.9 

44.9 

Total Work 

4.6 

7.7 

1.3 

15.6 

CPU Time 

7.9 

3. 1 

1.7 

5.5 

I/O Processor Time 

10.0 

13.4 

0.6 

8.3 

Data Channel Time 

1.9 

13.0 

-0.5 

32.3 

No. I/O Sequences 

1.5 

0.3 

-0.7 

-0.3 

Total Transfer Counts 

-0.2 

0.3 

-0.0 

-0.0 


distribution of device service time intervals can be displayed 
in addition to the determination of the mean and standard 
deviation of the distributions. 

One such use of the job scripts has been the generation 
of activity plots, indicating the patterns of file and device 
usage. Activity plots are similar in nature to page reference 
maps that depict a job’s memory demands. 5,25,26 Virtual time 
is displayed along the horizontal axis, and the various files 
and devices used by the job appear on the vertical axis. The 
time axis is divided into time intervals, and a vertical line is 
drawn in the interval for each file or device in use during 
the interval. Figures 3, 4, and 5 are examples of activity 
plots generated from the job scripts. The three entries la¬ 
beled “IOP ROU” are used for I/O sequences that do not 
contain file name entries and usually represent some super¬ 
visory function. The entry labeled “1AJ/1CJ” represents 
the I/O processor routines that advance a job to the next job 
step and handle job completion. The entry labeled 
“PFILES” is marked for any I/O sequence that uses a file 
associated with the permanent file system. The remaining 
names appearing on the Y axis are either system files or 
user files used by the job. The plots in Figures 3 and 4 are 
of the same job, executed in monoprogrammed and multi¬ 
programmed environments respectively. These activity plots 
offer visual evidence of the reproducibility of job scripts, as 
the two plots appear to be nearly identical. The activity plot 
in Figure 5 indicates that jobs may pass through several 
phases , with cyclic behavior within phases. This type of 
behavior appears to be analogous to the phase/transition 
behavior of several memory reference models being pro¬ 
posed. 26 Additional work is planned to investigate in more 
detail this phase behavior with respect to file and device 
usage. 

In addition to providing a means for investigating a job’s 
execution characteristics, the job scripts are also suitable 
for use in trace-driven simulation models. 19,20 A fairly de¬ 
tailed model of our system has been implemented using the 
scripts as trace data to drive the model. In order to validate 
the simulation model and the use of the job script represen¬ 
tation, a test workload of 16 jobs was selected. This work¬ 
load was executed on the CDC6500 system at Purdue, and 
job scripts were generated. Measurements of system per¬ 
formance were also taken using the software monitor and 
compared with the results from the simulator. Three quan¬ 
tities were chosen for validation: completion time of the 
entire workload, mean adjusted memory residency time, and 
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Figure 3—Activity plot of monoprogrammed job 


PASCLI 

LGO 

LANG 

VSIND 

-STACK 

XEQTIN 

SPL 


-XEQTM 

1 1 

OUTPUT 

1 

COMPIL 

1II 

SCR 1 

llllllllll 1 

SCR2 

I 1 

TEMP 

mi i 

INPUT 

ii 

IOP ROU 

{ ii i I 

IOP ROU 

ii it n i 

IOP ROU 

ii in ii ii i i 

1AJ/1CJ 

ii i i ■ 

CPU 

ilSIEMHEaiBBI ■ B 1 

0 

5000 


iiiiBuiiiiniiiiiiiiiii mi liiuiiiiiii iiiii mi ii in min 11 ibi 

ItlltllilUII 1 lilill 1II11 Hill 11 lit IIIIIII I II1111111 III 


HG20010 
FRAME 1 


I 

I I I 
I Bl I 


10000 15000 20000 25000 30000 

VIRTUAL TIME (MSEC) 


35000 


40000 


Figure 4 —Activity plot of multiprogrammed job 
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Figure 5—Activity plot showing phase behavior 


mean response time. Since all of the jobs were available for 
execution at the start of the experiment, the mean response 
time is just the average of the individual job completion 
times. It was not possible to duplicate in the simulator the 
exact scheduling algorithm used by the real system, so each 
job in the model was given an arrival time that corresponded 
to the time the job first started execution in the real system. 
The job script accurately reflects the resource demand for 
each device in the system, so device utilizations were not 
used for validation, since accurate modeling of the comple¬ 
tion time implies accurate modeling of the device utiliza¬ 
tions. 

In the initial test of the model, relative errors of 4.3 per¬ 
cent, 6.4 percent, and 10.6 percent were observed for the 
completion time, mean residency, and mean response times 
respectively. Subsequently the model was modified to use 
the file name and I/O operation entries to allow for CPU- 
I/O overlap on WRITE operations. After the modification 
was complete, the relative errors were 1.0 percent, 3.1 per¬ 
cent, and 6.7 percent. A summary of these results appears 
in Table VI. The number of job preemptions in each case is 
also included in the table, and it appears that the model does 
well in this respect also. Thus the job script representation 
can be used in a simulation model to approximate relatively 
closely the overlapped behavior in the real system. Further 


work is also planned on the use of the job script with trace- 
driven simulation models. 

Finally, the job scripts can also be used to determine the 
parameters for a central server model. 27 This is possible due 
to the basic structure of the job script, the CPU-I/O cycle. 
The transition probabilities are easily determined from a 
count of the number of intervals for each device, and the 
mean service times are easily obtained. We plan to produce 
a sequence of central server model parameters for a job, 
each set of parameters corresponding to a phase of the job. 
In combination with the memory requirement entries in the 
job script, it is hoped that these parameters can be used in 
hybrid simulation models 28 as well. 


TABLE VI.—Validation of Simulation Model 



Actual 

System 

Simulation Model 

Without 

Overlap 

With 
Over 1ap 

Completion time 1 

Mean residency time 1 
Mean response time 1 
Job preemptions 

1 Seconds 

119.886 

24.439 

53.295 

13 

125.101 
26.006 
58.922 
10 

118.709 

25.197 

56.869 

11 
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LIMITATIONS OF THE JOB SCRIPT 

The use of job scripts as a workload representation has 
some drawbacks. The major disadvantages are the need of 
a software monitor, the expense of producing the job scripts, 
and the lack of compactness in the representation. 

The use of a software monitor to gather the initial event 
data is an obvious limitation. The usual disadvantages as¬ 
sociated with the use of a software monitor are also implic¬ 
itly involved. Detailed knowledge of the operating system 
is required for the correct installation, operation, and main¬ 
tenance of the monitor. The monitor must also be modified 
whenever changes are made to the operating system. A 
program is required to analyze the event data produced by 
the monitor, and in the case of script generation, it must 
also produce the input required by the script generation 
program. 

The generation of the job script is also a somewhat ex¬ 
pensive process in terms of the computer time involved. 
Table VII lists the cost of each step in the script generation 
process for the 16 jobs executed in a monoprogrammed 
environment. Step 4, script analysis, is not part of the script 
generation process per se, but does provide detailed statis¬ 
tics about the scripts generated. The costs involved are 
higher when scripts are generated from data collected from 
the production system over a 20 minute period (in excess of 
$75.00 and over 1300 CPU seconds). Fortunately, the script 
generation process is a one time cost, and the scripts pro¬ 
duced can be used in a variety of ways. 

Another potential limitation of the job script representa¬ 
tion is that it is essentially just a record of job activity as 
observed by the system. For example, the job script’s per¬ 
ception of the I/O behavior of a program is obscured by the 
buffering of logical I/O activity (i.e., the software monitor 
detects I/O activity only when the buffer of the associated 
file is either empty or full). It is the authors’ contention 
however, that unless we are willing to instrument each pro¬ 
gram individually (or instrument the system I/O routines), 
we must be content with this “distorted” view of job be¬ 
havior. We also feel that since the job script is produced 
from information gathered at the point where a program 
interacts with the system, it should be a sufficient represen¬ 
tation for most applications. The job script representation 
as described appears to be suitable for use only with non¬ 
virtual memory systems. It is possible however that it could 


TABLE VII.—Cost of Producing Job Scripts 


Job Step 

Time 1 

I/O 

Units 2 

Cost 3 

Size of 
File Created 4 

Software Monitor 

13.019 

3660 

$1.22 

112,501 

Data Analyzer 

101.952 

5788 

$5.22 

139,756 

Script Generator 

35.168 

2255 

$1.84 

36,480 

Script Analysis 

8.019 

648 

$0.54 

— 


Observation interval covered 449.371 seconds 
Scripts for 16 jobs were produced 


l CPU seconds used (including time to load the program) 
2 one I/O unit = 1000 characters of data transferred 
3 as computed by the Purdue University Computing Center 
4 number of 60-bit words 


be adopted for use with a paged virtual system. At the very 
minimum, additional information concerning the page ref¬ 
erencing behavior of a job, along with how it relates to CPU 
usage would be needed. This is one possible avenue for 
future research. 

Finally, the inclusion of every resource interval in the job 
script leads to a lengthy representation of a job. Table VII 
also includes the size of the file generated by each step in 
the process. Although the script generator reduces the size 
of the initial event data file, the script file is still sizable (in 
excess of 36,000 words to represent approximately 500 sec¬ 
onds of job activity). Work is currently under way to find 
smaller representations using the job script as a starting 
point. This involves the determination of the “important” 
characteristics of a job, and the elimination of unneeded 
information. 

CONCLUDING REMARKS 

A job script, defined in this paper to be a detailed list of a 
job’s demands for system resources, does form an accurate, 
reproducible, and useful workload description at the job 
level. In turn, collections of scripts can be used to com¬ 
pletely describe the workload for a system. The level of 
detail present in these scripts is both a strength and a weak¬ 
ness of this descriptive form; on the plus side, the detail 
present means that few assumptions are required when using 
job scripts (all relevant information is present); on the minus 
side, the volume of information makes it difficult to form 
overall conclusions and generalizations from the scripts. In 
a very real sense, you cannot see the forest for the trees. 

At Purdue, we have an ongoing project into the develop¬ 
ment and validation of computer system models. A critical 
facet of this project is the formulation of techniques which 
can be used to describe a system workload. Job scripts have 
emerged as one such technique. Future work will focus on 
further reduction in the scripts, perhaps along the lines sug¬ 
gested by the phase-cyclic behavior patterns noted above. 
The goal would be the automatic generation of job descrip¬ 
tions and then system workload descriptions which are not 
only accurate, reproducible, and useful, but also compact. 

Another line of investigation which could be pursued is 
the reduction of job scripts so that the resulting information 
could be used to accurately and succinctly characterize a 
system workload in a form that would assist computer sys¬ 
tem managers. For example, such descriptions could be 
essential factors in upgrading existing systems and procure¬ 
ment of new systems. While using job scripts in this fashion 
can only be conjectured presently, it can be stated that the 
scripts in their present form do contain the necessary infor¬ 
mation. The problems lie in the reductions and final form 
that would be required. 
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INTRODUCTION 

It is a commonly observed phenomenon that the workload 
on a computer system has a tendency to increase, resulting 
in poorer service or the aquisition of new, more powerful 
computers. Since the procurement of a large scale computer 
is a very lengthy process, it becomes crucial to predict the 
workload a system is likely to have several years into the 
future. 

Most of today’s general purpose third generation com¬ 
puters are used in a multiuser, multiapplication environ¬ 
ment. The workload handled by a computer is a composite 
of the computing needs of all of the users; each putting 
different resource demands on the systems. Frequently, the 
load increases as new applications are put on the system, or 
as users find new ways of making use of the facilities of the 
system. A commonly used practice is to describe the com¬ 
posite workload in terms of measures such as total CPU 
hours used, the number of terminal connect hours, or the 
number of transactions handled. While such measures give 
an idea as to the load a system handles, they are rarely 
adequate for prediction purposes. 

The major, if not the only, purpose of a computer instal¬ 
lation is to meet the computing needs of its user population. 
A computer system is used for different applications, the 
nature, volume, and characteristics of which change with 
time. It is these changes that result in a change in the work¬ 
load on a computer system. But as the changes in different 
applications may be different, such changes may not be 
visible from the composite measures. Therefore, the work¬ 
load prediction techniques have to take into account each 
of the components of the workload. In this paper we discuss 
some techniques which are applicable to the workload pre¬ 
diction problem. 

Before we can consider the techniques for predicting the 
workload of a computer, we need techniques for character¬ 
izing the current workload of a system. In the second section 
we review a workload characterization technique which uses 
clustering to uncover the natural groupings in the workload. 
In the third section we discuss the applicability of some of 

* This research was supported in part by the National Aeroanutics and Space 
Administration, Goddard Space Flight Center, under Grant NASA #NAS 5- 
24092, and in part by the Environmental Protection Agency under Grant 
R805478-01-0, to the Department of Computer Science, University of Mary¬ 
land, College Park, Maryland. 


the statistical forecasting techniques to the workload pre¬ 
diction problem. In the fourth section, we discuss the ways 
in which the cluster description of the system’s workload 
may change and the ways in which such a description may 
be used to predict future workloads on the system. The fifth 
section of this paper presents an example of a workload 
prediction study that was carried out using the techniques 
discussed in the fourth section. The system described in the 
fifth section is currently being used for both program de¬ 
velopment and production work. 


THE CLUSTERING APPROACH TO WORKLOAD 

MODELLING 

If a computer system were to be used to carry out a small 
number of tasks repeatedly, then it would be a rather 
straightforward task to create an accurate, validated model 
of the workload. But, for a general purpose computer system 
that is shared by a large user population creating an accurate 
workload model is not easy. 

In a multiuser environment each user makes a sequence 
of computational requests to the system. While the request 
patterns of one user are likely to be independent of the other 
users present on the system, the time instants at which the 
requests are made may depend on the system response char¬ 
acteristics, which in turn, depend upon the actions of all of 
the other users. No reliable models to date have been for¬ 
mulated for the behavior of a user. One often has to infer 
the behavior of the user based upon the data collected by 
the system on logs. 1 In its raw form, this information tends 
to be so voluminous that it can hardly be used directly as a 
workload model. Therefore, a simplified model has to be 
created. The problem of formulating accurate workload 
models then reduces to the problem of finding natural pat¬ 
terns or structures in the huge amount of data available from 
a system log. 

The problem of finding structures in large volumes of data 
has often been addressed in the field of pattern recogni¬ 
tion. 2,3 As was noted in Reference 4, however, there are 
some very basic differences between the pattern recognition 
problem and the workload characterization problem. Pri¬ 
marily, the pattern recognition problem assumes the exis¬ 
tence of well defined classes, or groupings, in the observed 
data. The groupings we are attempting to find in the work- 
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load are strictly for the purposes of modelling the workload 
and no a priori groupings can be inferred. Several of the 
pattern analysis techniques can, however, be used for work¬ 
load modelling. In particular clustering has been used suc¬ 
cessfully for creating workload models. 4-9 

In using clustering to model the workload, first we select 
the degree of aggregation by defining the lowest level of 
detail that we plan to model as a “workstep.” Next we 
describe a workstep as a point in an appropriately defined 
multi-dimensional space whose axes represent resource 
usage levels, or other features of a workstep. Next, a set of 
worksteps is observed on the system and clustered to un¬ 
cover their natural groupings. 

While the validity of cluster based workload models has 
been well established in several studies cited above, the 
groupings are shown to change by changing the feature set. 4,6 
This is quite natural because no a priori groupings exist and 
thus changing the feature set changes the perspective of the 
problem. This gives the analyst some flexibility in producing 
good models by choosing appropriate feature sets. 

As noted in the first section, a problem with workload 
prediction arises from the fact that different groups of ap¬ 
plications are likely to change in different ways. By using a 
cluster based workload model we take into account the 
characteristics of such individual groups. Before discussing 
how we can use such models in prediction let us discuss 
some of the statistical forecasting techniques in terms of 
their relationship to the workload prediction problem. 

TYPES OF CHANGES OF THE WORKLOAD 

Frequently the statistical forecasting techniques 10 re¬ 
ported in the literature address themselves to the forecasting 
of a single variable. The workload of a computer system can 
hardly be considered univariate. However, for the purposes 
of discussion we are going to assume that a single unit of 
work can be defined and that we can talk about such things 
as the load on a system doubling, or varying in some way, 
and that we can plot the workload vs time on a graph. 

The statistical forecasting literature discusses four basic 
types of changes that may occur and methods by which they 
may be predicted. Let us now examine these four types of 
changes and discuss them in terms of the workload predic¬ 
tion problem. 

First, there may be a random stochastic variation in the 
amount of work performed over a period of time. Second, 
the amount of work may be a monotonically increasing or 
decreasing function of time. Third, the amount of work 
performed may show an oscillatory nature, that is there may 
be some type of periodic behavior. Finally, there may be 
abrupt discontinuities in the workload. Of course combina¬ 
tions of these may also occur. Let us consider each of these 
changes briefly. 

Random variations 

As we noted above, on a system which supports a large 
user community the workload observed on the system is the 


composite of the workload generated by individual users. 
As the independent users may not follow any particular 
patterns in submitting their processing requests, a sequence 
of requests from a user may be treated as a stochastic point 
process. 11 The system workload may, therefore, be obtained 
as a superposition of the individual point processes and may 
be modelled as a random point process. The exact nature of 
the point process depends on the size of the user population 
as well as the characteristics of the individual users. 

In addition to the point process type of characteristics of 
the workload, the amount of work to be performed in re¬ 
sponse to each request may also show a random variation. 
If we take, for instance, the case where the machine is used 
primarily for program development, we would typically ex¬ 
pect to be able to describe the load supplied by an individual 
by a pair of random variables drawn from two, possibly 
different, distributions. First, there is the random factor 
determining how often the user submits a job. This will 
depend upon, among other things, how long it takes him to 
fix any bug found in his previous run. There will also be a 
random factor in the amount of work that the next job will 
perform, which will depend upon the program being devel¬ 
oped and the type of bug that next appears. Therefore, some 
random variations in the load on the system seem quite 
reasonable. 

Due to the randomness of such variations they can only 
be characterized by describing their probabilistic properties. 
Very little can be done with respect to forecasting them 
except to forecast the parameters of their probabilistic de¬ 
scriptions. The distributions of the loads may be useful in 
determining peak loading conditions. 


Monotonic changes 

By a monotomic change in the workload we mean that the 
amount of work performed by the computer system shows 
a steady change over time. This change may be observed as 
a trend and may reflect an increasing or a decreasing work¬ 
load. In practice, however, the workloads of computer sys¬ 
tems have a tendency to increase rather than decrease. The 
trend shown by such changes may follow any pattern e.g., 
straight line, quadratic, exponential. 

There are several physical justifications for the fact that 
loads on computer systems tend to increase with time. First 
many programs that have been in use for a long period of 
time tend to grow larger and slower as changes are made or 
new features are added. Second, many programs that have 
running times proportional to the number of records proc¬ 
essed tend to grow longer becasue each subsequent run is 
required to process a larger number of records. Third, many 
installations that perform program development or program 
maintenance seem to enlarge their staffs and thus present a 
larger processing load to the central computer system. 

As the monotonic changes are likely to follow well defined 
trend lines they lend themselves well to forecasting. Under 
the assumption that the trends do not change, a prediction 
of the workload can easily be made by following the trends. 
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Oscillatory loads 

The workload on a system may show a periodic behavior, 
increasing and decreasing in an oscillatory manner. We may 
argue that this is the most frequently observed behavior of 
computer system workloads. While a computer system may 
run 24 hours a day, the loads go up and down depending on 
the hour of the day. This variation may be connected with 
the user work hours and work habits. Similar types of os¬ 
cillatory behavior with different periods may have a number 
of causes. Many installations have tasks which are per¬ 
formed daily, weekly, biweekly, monthly, quarterly, or 
yearly. 

Another example of this phenomenon can be observed at 
a typical university computer center where the period of the 
oscillation is one semester. At the beginning of the semester, 
the load is comprised of the research users running relatively 
large jobs. As the semester continues the load gradually 
changes until by the end of the semester the load is com¬ 
prised of a large number of students running small jobs. 
Also, as more projects are due at the end of a semester the 
load increases significantly at that time. 

As the system workload is a composite, it may contain 
components with different periodic behaviors. For predict¬ 
ing the workload at some future time we need to identify 
each oscillatory component and its period. Spectral decom¬ 
position techniques may be used for this purpose. 

Discontinuities 

When one plots a graph of the amount of work submitted 
versus time one may find points of discontinuity in the curve 
or its derivative. These discontinuities correspond to sudden 
changes in the load on the system. Such abrupt changes may 
occur for any one of several reasons. 

A frequently observed cause of sudden changes in the 
load on a computer system are changes in its normal oper¬ 
ating procedures. Examples of such changes in operating 
procedures would include switching major files from tape to 
disk, or switching major applications from batch to time 
sharing. Both of these changes are likely to introduce rather 
large changes in the system’s load. In an environment in 
which separate systems are used for development and pro¬ 
duction, the changes in the production machines’s load will 
tend to be very abrupt as applications make the transition 
from the test phase to the production phase. 

If a new application is decided upon and a significant 
number of new programmers are hired to perform the im¬ 
plementation, then one would expect to see the derivative 
of the graph of load change abruptly and expect to see the 
load increase rather quickly there after. 

Note that abrupt changes causing the discontinuities are 
a result of external factors and, therefore, cannot be pre¬ 
dicted on the basis of information available from within a 
system. For the models created on the basis of system meas¬ 
urements such changes have to be treated as random phe¬ 
nomenon. Since these changes may affect the loads drasti¬ 
cally they have to be taken into account in any workload 


prediction. However, since these abrupt changes are due to 
external factors it may be possible to predict them by ex¬ 
amining the external factors. The quantitative impact of 
these abrupt changes may be less precisely known due to 
the fact that they will occur in the future and thus cannot be 
measured. 

Composite changes in the load 

In the discussion above we observed that all four types of 
changes are likely to occur in a computer system workload. 
Examining the causes of such changes we note that they are 
not mutually exclusive and that the workload changes ob¬ 
served on real systems can only be explained in terms of 
two or more of these. For workload prediction we have to 
identify these component change behaviors first. 

As the system workload may consist of several segments 
it is not necessary that all segments of the workload will 
exhibit similar changes in behavior. In fact, we expect that 
the change in the behavior of individual components of the 
workload will be much simpler than changes in the behavior 
of the overall workload. A way of identifying the segments 
of the workloads is to use the cluster based characterization. 


THE CLUSTER DESCRIPTION OF THE WORKLOAD 

In this section we consider the ways in which the cluster 
description of a computer’s workload may possibly change 
with time and how the different types of changes in the 
workload will be reflected in its cluster description. 

Predicting changes in the cluster description 

If one is using the clustering approach to describe the 
system workload, the problem of predicting a future work¬ 
load on the system corresponds to the problem of finding 
the cluster description for what the workload will be at some 
point in the future. As stated earlier, the purpose of clus¬ 
tering the workstep population is to obtain a homogeneous 
group of workstep s that behave in the same manner. Since 
the total system workload would be composed of the sum 
of the work supplied by each cluster one would expect to 
find the future system workload to be composed of the sum 
of the work supplied by each cluster after the appropriate 
modifications have been made to the cluster. Therefore, to 
find the predicted load on the system we must examine each 
of the clusters that are present in the system workload and 
determine how each of these clusters will change. We then 
make the appropriate modifications to the cluster descrip¬ 
tions and from these descriptions find the description for the 
entire system load. This operation is likely to be easier and 
more accurate than trying to deal with the entire workstep 
population at one time. These operations do of course re¬ 
quire a reather detailed knowledge of the membership of 
each cluster as well as knowledge of how these worksteps 
have been varying in the past and the types of variation that 
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these worksteps are likely to exhibit in the future. Basically, 
all cluster analysis allows us to do is to model small segments 
of the load independently and gives us ways of combining 
these models into a model of the total system workload. The 
prediction of these changes may be accomplished using 
some of the techniques described earlier. 

Changes in the cluster description of the load 

A cluster taken as an abstract entity only has a limited 
number of properties that may vary. A cluster can be 
thought of as having a location in an appropriate n-dimen- 
sional space and can move or drift in this space. A cluster 
has members, that is, it is composed of some worksteps 
from the total workstep population and therefore some 
things can be said about the distribution of points in the 
cluster. Finally, a cluster can be used to describe some 
segment of the total system workload. In addition to the 
information available about the cluster itself we can analyze 
the members of the cluster and determine any additional 
information that we need to know such as their arrival rates 
or what types of work they performed. 

The cluster description of the total system workload is 
composed of a group of clusters obtained from clustering 
the workstep population. In addition to the description of 
the individual clusters as described above we also know the 
proportion of the total system load attributable to each clus¬ 
ter. 

Before discussing the ways in which the cluster descrip¬ 
tion of the workload may change, let us consider what it 
means for a given cluster to appear in two different cluster 
runs. If the system is running in a production environment 
in which the same programs are run repeatedly then two 
clusters obtained by the analysis of data from two different 
times are considered to represent the same cluster if the 
cluster obtained from the second run contains most of the 
members of the cluster obtained from the first run. In a 
nonproduction environment, where different programs are 
run each day, this method of identifying clusters cannot be 
applied. Here we have to ascertain what type of job consti¬ 
tutes the cluster in question and to find the similar type of 
cluster in the output of the later cluster run. For example, 
in a university workload one would expect the amount of 
resources used by small student programs to grow with time. 
Therefore, if one found a cluster both from the beginning of 
the semester and a cluster from the end of the semester 
which contained most of the jobs from the introductory 
programming class then one would say the two clusters 
correspond even though no single job appears in both clus¬ 
ters. 

Since a cluster has a position or location in an n-dimen- 
sional space its location may change with time. This would 
occur if most of the worksteps which made up the cluster 
changed their feature values in approximately the same way. 
If this were the case, and some of the worksteps did not 
change in the same manner as the rest of the worksteps then 
those worksteps would move with respect to the cluster 
mean. They may in fact move far enough away from the 


cluster mean so as to be eliminated from the cluster entirely. 
When the mean of a cluster moves it is quite likely that 
some points not previously in the cluster would become 
close enough to the cluster mean to be absorbed into the 
cluster. Thus, when a cluster drifts it is not unusual to find 
that some old members leave the cluster and some new 
members are added to the cluster. This type of change in 
the cluster description would of course correspond to the 
case where there was some monotonic change in the amount 
of system load, that is whether some or all of the worksteps 
originally contained in this cluster increased in size. 

If a cluster contains worksteps which change in different 
ways over a period of time, the cluster may fall apart. If this 
happens the worksteps which were in this cluster may drift 
close enough to other clusters and join them. As a result the 
percentage of the total system workload attributable to each 
cluster may change also. 

Another type of change in the workload that may change 
the cluster description of the workload is the introduction 
of new applications to the system, or changes in the oper¬ 
ating mode of existing applications. In many cases the 
easiest way to handle such predicted changes is to treat the 
new or changed application as if it were a cluster by itself. 
This may be necessary since one might have to estimate the 
resource requirements of the new application as no measued 
data may be available. Therefore, one might want to handle 
this cluster in a different manner than the clusters that were 
obtained by observing the load on the system. 

The applications that already exist but are having their 
operating mode changed, such as being changed from batch 
to timesharing or from tape to disk, could be treated the 
same as entirely new applications except that somewhat 
better estimates may be made for their resource require¬ 
ments. These worksteps can be handled by removing them 
from the cluster to which they belonged and treating them 
as if they were a separate cluster by themselves. 

An oscillatory change in the amount of work performed 
by the system, or a random variation in the amount of work 
performed by the system, would typically not involve a real 
change in the description of the clusters themselves, but 
rather would involve a change in the number of points as¬ 
signed to each cluster. Therefore, the variation in the load 
would be primarily attributable to changes in the arrival 
rates for the worksteps belonging to the various clusters. 

It should be noted that the above techniques need only be 
applied if a substantial portion of the load is expected to 
change. If only a few small worksteps are expected to 
change then the effect on the total system load may be 
extremely small and can be isolated to one or two clusters. 

Predicting changes in applications 

An application which continues to be processed over a 
period of time is often used to process transactions or rec¬ 
ords of some type. While the computer resources that will 
be required may not be easily predicted, in many cases an 
accurate prediction can be made of the number of transac¬ 
tions, or the number or records that will be processed as 
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these numbers are tied to external factors that may be pre¬ 
cisely known or easily predicted. For example, the rate of 
increase of a payroll processing program depends on the 
rate of increase in the size of the employee records file, 
which in turn depends on the rate at which the employment 
in the company is going to change. Changes in employment 
levels are usually controlled by the management and can be 
used to project the resource requirements of the payroll 
program. 

In cases such as this, a far more accurate prediction of 
the future systems load may be made by first forecasting the 
changes that will occur at the applications level and then 
mapping these changes into changes at the resource usage 
level. In some cases this method will be the only way in 
which reasonable estimates may be made of the future sys¬ 
tems workload. 

When an abrupt change occurs in the load the changes are 
almost always attributable to external factors such as 
changes in budget levels, the introduction of new applica¬ 
tions, or other factors that are clearly known in advance. In 
these cases the changes in the computer load must also be 
predicted at the applications level and then the expected 
computer load must be determined based on these predic¬ 
tions. 

Clearly the accuracy of the predictions of the applications 
workload will depend on the accuracy with which the influ¬ 
ences of the external factors can be determined. 


AN EXAMPLE 

In this section we present an example of how the tech¬ 
niques described in the preceding section can be used to 
predict the load on a real system. One of the difficulties in 
describing the results of a prediction study is that the ac¬ 
curacy of the prediction may not be determined for several 
years. Therefore, the importance of this example is not that 
it purports to be a complete prediction of the load on a given 
system, which it is not, but rather that it serves to illustrate 
the techniques that were described in the preceding sections. 

The data used in this example was drawn from work done 
predicting the load for an installation that operates two IBM 
370/158 computers. These machines, which are used for 
program development and the execution of several produc¬ 
tion systems, process approximately 20,000 job steps per 
month. This workload prediction study was performed be¬ 
cause a need for a replacement for the two 370/158’s had 
been identified and it was necessary to produce a benchmark 
to aid in the procurement of this replacement system. The 
new system was to be sized such that it would satisfy the 
needs of the installation for a five to eight year period. 
Therefore, a benchmark had to be produced to represent the 
load five years in the future. 

This installation was chosen as the example for this paper 
because the load shows many of the types of variations 
discussed above. First, the basic data set that is used by 
some of the production systems will be moved from tape to 
disk in the near future. In addition, one of the production 
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Figure la—Present cluster 

systems will soon have its operating mode changed from 
batch to timesharing. The program development portion of 
the workload can be expected to increase due to the modi¬ 
fications and enhancements being made to the production 
systems. 

Seven features were extracted for each of the job steps 
and the job steps were clustered using the techniques in 
References 4-6. This clustering produced twenty clusters, 
ranging from a cluster of thirty executions of the same work- 
step to a cluster of over 10,000 relatively small worksteps. 

Rather than presenting data on the twenty clusters and 
over 20,000 data points in a seven dimensional space, the 
discussion below will deal with only a few of the clusters 
and some representative data points. In addition, the data 
presented in the example is presented in terms of only two 
features rather than the original seven features. We feel that 
by presenting the data in this manner we will be better able 
to illustrate the techniques involved than if we had attempted 
to give a detailed description of all of the data. 

The scatter plot of Figure la represents a cluster of work- 
steps from two distinct classes of production systems. For 
the purposes of this discussion only two features, tape 
EXCP’s and disk EXCP’s are shown. The worksteps in the 
two classes perform essentially the same task, that is editing 
input data and processing updates to a large master file. 
There is no way, based upon their resource requirements, 
that the worksteps from these two classes can be distin¬ 
guished. Therefore, all of the worksteps are placed into one 
cluster. The worksteps are described as coming from two 
distinct classes because the worksteps in class 1 are from a 
production system that will have its master file moved from 
tape to disk, while the master file used by the worksteps in 
class 2 will remain on tape. Therefore, the types of resources 
required by the worksteps from this class will change rather 
abruptly. Figure lb shows a scatter plot of the clusters after 
predicting the future resource requirements of these work- 
steps. Since the resource requirements of the worksteps in 
class 2 are expected to remain essentially constant they are 
plotted in the same position. The worksteps in class 1 are 
presented as a new cluster with a significantly lower number 
of tape EXCP’s and a correspondingly higher number of 
disk EXCP’s. These plots are not meant to imply that there 
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Tape EXCP's 

Figure lb—Predicted clusters after splitting of single cluster 


will be a one for one conversion of tape EXCP’s to disk 
EXCP’s, but rather that most of the I/O that these worksteps 
perform will be performed to disk. The exact number of disk 
EXCP’s executed will depend upon a number of factors 
such as the buffer size that is used, the disk access method 
that is used, expected changes in the size of the master file, 
and changes in the number of records to be processed by 
each workstep. Each of these factors will influence the num¬ 
ber of disk EXCP’s that a workstep will perform. The num¬ 
ber of records to be processed by each workstep may change 
for several reasons. First, the system will become more 
heavily used due to increases in the application load. This 
would increase the amount of I/O performed per day. On 
the other hand, once the data file is placed on disk, and is 
thus more accessible, one might decide to run a larger num¬ 
ber of worksteps each of which processes a smaller number 
of records, thereby decreasing the number of disk EXCP’s 
per workstep. Each of these factors depends upon factors 
to be decided upon before the system conversion can be 
made and thus can be used in predicting the system load. 

The processing load at this installation is subject to a 
variety of external influences. The processing carried on at 
this installation is used to support the activities of a large 
number of divisions throughout the parent organization. 
Therefore, the processing load on the computer will be ef¬ 
fected by management decisions on the initiation or curtail¬ 
ment of activities throughout the organization. In fact, a 
recent decision by management to reduce a specific activity 
of one of the divisions will have the effect of reducing the 
number of records processed by one of the application pro¬ 
grams by approximately one third. There is no way that 
such a change in the processing requirements of the pro¬ 
duction system could be predicted from data observed on 
the machine. However, the external factors influencing this 
change are well understood and their effects can be very 
accurately determined well in advance. Use of this external 
information is in fact the only way in which the load on the 
system due to this application can be accurately predicted. 

Figures 2a and 2b represent a cluster that contains many 
of the program development worksteps. This cluster is com¬ 
posed of a large number of worksteps drawn from a number 
of sources. Based on observations carried out over a period 
of time it has been observed that the amount of load attrib¬ 
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Figure 2a—Present program development cluster 


utable to the program development cluster has been increas¬ 
ing. This increase in load is due to an increase in the number 
and size of the program development worksteps. 

This cluster’s size and position can be predicted using 
trend analysis. This is a case in which the position of the 
cluster drifts due to the movement of most of the data points 
in the cluster. This movement can be seen when the clusters 
in Figures 2a and 2b are compared. The cluster in Figure 2b 
contains more points that the cluster in Figure 2a. This is 
due to the expected increase in the number of program 
development worksteps that will be executed by the system. 

It should be noted that as the cluster mean drifts and the 
size of the cluster increases the variance of the data points 
in the cluster increases as well. This occurs because not all 
of the worksteps are expected to change in precisely the 
same manner causing the data points to become more spread 
out in the seven dimensional space. 

When observing this cluster over a long period of time, it 
becomes clear that the trend is definitely one of constant 
growth in terms of both the number of worksteps and the 
size of the worksteps. This is not meant to imply that the 
rate of increase is absolutely constant. There is some ran¬ 
dom variation in the number of program development work- 
steps executed in a given time period. However, this type 
of random variation is commonly encountered in trend anal¬ 
ysis studies and is easily handled. 

There is also a rather strong oscillatory component to the 
workload. While some of the production systems are run on 
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TABLE I.—Amount of Load Attributable to Various Clusters 


Cluster 

Percent of 
workload 

normal environment 

Percent of 
workload 

periodic environment 

1 

51 

10 

2 

10 

2 

3 

15 

1 

4 

1 

84 


a daily basis, others are run only once every two weeks. 
The worksteps from these systems tend to be clustered 
together due to their rather peculiar I/O requirements. Since 
the work that these periodic production systems perform is 
quite time critical they are typically run in stand alone mode 
so as to insure their timely completion. Therefore, there is 
a cluster that contributes virtually none of the load most of 
the time, but every two weeks contributes essentially all of 
the load. The amount of time required to complete these 
worksteps is somewhat variable due to such factors as sys¬ 
tem crashes. Table I contains the percentage of the load 
contributed by some of the clusters. The two sets of data 
are for two separate time periods, one which contains the 
time period in which these periodic systems are running, 
and the other when the system was running what might be 
considered its “normal load.” The percentage of the load 
contributed by the non-periodic worksteps is non-zero in the 
column for the periodic work as the period for which the 
data was collected overlaps the execution of the periodic 
systems. 

When the set of benchmark programs and the system’s 
requirements are released to the vendors there will be two 
benchmark experiments to be run. First, a set of benchmark 
programs representing the normal processing load on the 
system will be run and the results compared. Second, there 
will be a requirement that the system to be purchased must 
be able to execute a representative portion of one of the 
periodic worksteps in less than a specified amount of time. 
This has to be done because the system has two operating 
modes and a new system must be able to handle two very 
distinct types of loads. It is quite likely that if the worksteps 
from the periodic systems were combined with the rest of 
the worksteps into one benchmark then the results that 


would be obtained from such a benchmark would most likely 
be rather inaccurate. 

CONCLUDING REMARKS 

In this paper we consider the problem of workload predic¬ 
tion and note that an accurate prediction can only be made 
by considering the changes in the individual segments of the 
workload. For this purpose the segments, or natural group¬ 
ings, of the workload may be identified using clustering, and 
then the way components, or clusters, are going to change 
can be taken into account in the prediction. 

The future workload is often affected to a significant de¬ 
gree by external factors. It is crucial that such external 
factors be taken into account when making the workload 
predictions by considering the new applications or changes 
to the modes of operation as well as the effects of external 
factors on the applications. 
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INTRODUCTION 

The schedulers of nonpreemptive batch processing systems 
are often based on the first-come first-serve (FCFS) or short- 
est-job-first (SJF) algorithms. The former is least discrimi¬ 
nating in the sense that it does not base its scheduling de¬ 
cisions on the service requirements of jobs. In the SJF 
scheme, the service requirement of a job (or at least an 
estimate for it) must be known at the time of arrival. It is 
used to assure good response for short jobs by discriminating 
against long ones. Overall, the system response for SJF is 
better than that for FCFS, but the penalty imposed on long 
jobs is excessive. Brinch Hansen 1 suggested an alternate 
scheme which improves the overall response characteristics 
of FCFS without excessively delaying long jobs. In this 
scheme, the highest response-ratio next (HRN) scheme, the 
priority of an arriving job is set to zero. It then increases 
linearly in time at a rate inversely proportional to the service 
requirement of the particular job. When a job departs, the 
priorities of all jobs in the queue are evaluated and the one 
with the highest priority starts to execute. 

The expected waiting time (the time between arrival and 
start of execution, also called the delay ) of a job conditioned 
on its service requirement is a common measure for the 
performance of a scheduling algorithm. We call this condi¬ 
tional expectation the delay function W(x), where the pa¬ 
rameter x denotes the service requirement. The response 
time of a job is the sum of its delay in the queue and its 
service requirement, and the response-ratio is defined as the 
response time divided by the service requirement. Note that 
the response-ratio is equal to one plus the delay divided by 
the service requirement. Overall, the rate at which a job 
attains service is equal to its service requirement divided by 
the response time. Thus, its processing rate is equal to the 
inverse of the response-ratio. By selecting the job with the 
highest response-ratio for service, the HRN discipline at¬ 
tempts to service competing jobs at equal rates. To quantify 
the behavior of the HRN scheme, Brinch Hansen derived 
an approximate expression for its delay function which is 
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valid for a system with Poisson job arrivals and hyperex- 
ponentially distributed service times. 1 A comparison of this 
approximation with the delay functions of FCFS and SJF 
demonstrates how HRN strikes a balance between these 
two extremes. In this paper, we derive the exact HRN delay 
function for M/G/l systems, i.e., single-server systems with 
Poisson job arrivals and arbitrarily distributed service times. 
The derivation amounts to solving an integro-differential 
equation which was first developed in Reference 1. Making 
explicit use of random variables, we proceed to formulate 
this integro-differential equation via a slightly different 
method. 


FORMULATION OF THE INTEGRO-DIFFERENTIAL 

EQUATION 

According to the classification scheme of Ruschitzka and 
Fabry, 7 a scheduling algorithm is specified in terms of a 
priority function, a decision mode, and an arbitration rule. 
The priority function of the HRN scheme is 

P(r,x)=r/x (1) 

where the job parameters r and x are the time a job has 
spent in the system and its service requirement respectively. 
The decision mode is nonpreemptive, i.e. scheduling deci¬ 
sions are only made when a job departs or when an arriving 
job finds the system empty. A first-come first-serve arbitra¬ 
tion rule will be used to resolve conflicts among jobs with 
equal highest priority. 

Jobs arrive at the M/G/l system at a rate X. Their service 
times S are assumed to be mutually independent and iden¬ 
tically distributed. g{s), ES, and ES will denote the proba¬ 
bility density function, the expectation, and the second mo¬ 
ment of S respectively. The first two moments are assumed 
to be finite. A job with a service requirement of S=s will be 
called an s-job. p=XES stands for the system load which 
must be less than one. We express the delay and other 
random variables of interest for a particular, but arbitrary, 
job which we call the test job. Its service requirement will 
be denoted by x. For the averages of waiting times we shall 
use time averages which, due to the assumption of Poisson 
arrivals, are identical to job averages. 
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The delay of the test job can be equated with the amount 
of service performed on other jobs. Using this approach (cf. 
Reference 6), we distinguish between two groups of jobs: 
the early arrivals which arrive before the test job and the 
late arrivals which arrive after it. Similarly, we define the 
average early work W E (x) (late work W L (x)) as the total 
amount of service performed on all early (late) arrivals dur¬ 
ing the test job’s lifetime in the system. With these defini¬ 
tions, the average delay of the test job is given by 

W(x)=W*(jc)+W l (jc). (2) 


We proceed to further decompose the term for the early 
work. 

When the test job arrives, the system is either empty (idle) 
or servicing an early arrival. We denote the average remain¬ 
ing service of this early arrival by W 0 . More early arrivals 
may be waiting in the queue. An 5-job in the queue with 
s<x will be serviced before the test job since its priority 
grows faster according to equation (1). The average cumu¬ 
lative service of all these early 5-jobs with s^x will be 
denoted by W i (x ). An 5-job with s>x will only be serviced 
before the test job if it has been in the system sufficiently 
long such that its priority exceeds the faster growing priority 
of the test job. We call the average cumulative service of 
these jobs the work W 2 (x), i.e., the work associated with 
those early arrivals that get serviced before the test job 
although their service requirement is higher. In sum, the 
average early work encountered by the test job is comprised 
of three different types: 

(jc)= W 0 + W 1 (jc)+ W 2 (jc) . (3) 


The average time which the test job must wait until the 
job (if any) which it finds in service completes its remaining 
service is well-known (c.f. Reference 2, p. 16): 

W 0 =XES 2 /2. (4) 


In order to obtain expressions for W t (x) and W 2 (x), we first 
establish an expression for the average number of 5-jobs in 
the queue. Applying Little’s theorem 3 to 5-jobs in the queue, 
the average number n(s)ds of 5-jobs which the arriving test 
job finds in the queue is simply 

n(s)ds=XW(s)g(s)ds. (5) 

Furthermore, the average cumulative service requirement of 
all 5-jobs in the queue is sn(s)ds. Since 5-jobs with 5<x will 
be serviced before the test job, their contribution to the 
early work is 


J r a: r x 

5tt(5)e?5=A sW(s)g(s)ds. (6) 

o •'o 


Depending on the time it spent in the queue, an 5-job with 
s>x may or may not be serviced prior to the test job. 
Consider the limiting case in which the priorities of an early 
arrival and the test job are equal at the instant at which the 
server begins to work on the early arrival. Assuming that 
the 5-job arrived T seconds before the test job and that it 
has waited V seconds, the test job will have spent V-T 
seconds in the system. We now equate their priorities ac¬ 


cording to equation (1) to get 

V/s=(V—T)/x. 

Solving for T, we obtain 

T=V(1-jc/5) (7) 

as the minimum time between the arrivals of the 5-job and 
the test job if the 5-job is to be serviced before the test job. 
Conversely, all 5-jobs with s<x that arrive within T seconds 
prior to the test job’s arrival will be serviced after the test 
job. Due to the assumption of Poisson arrivals, the average 
number l(s)ds of these latter 5-jobs is equal to the rate 
Xg(s)ds of their arrival multiplied by the expected length of 
T: 


l(s)ds=\g(s)E[T]ds . 

After substituting T from equation (7) and using the equality 
E[V]=W(j) which follows from the definition of the delay 
function we get 

l(s)ds=\(l~x/s)W(s)g(s)ds. (8) 

The average number n(s)ds of 5-jobs in the queue is, of 
course, the sum of the average number l(s)ds of those which 
get serviced after the test job and some average number 
e(s)ds of those which are serviced prior to the test job. 
Thus, we have e(s)ds=n(s)ds-\(s)ds, and substituting 
from equations (5) and (8) yields 

e(s)ds=\(x/ s) W(s)g(s)ds. 

The average service required by all 5-jobs is s times their 
number, and the average cumulative service of all early 
arrivals which require s>x seconds of service and get ser¬ 
viced before the test job is therefore 




W(s)g(s)ds. 


(9) 


The early work W B (*) in equation (3) is now completely 
determined by equations (4), (6), and (9). 

We now proceed to evaluate the late work. Recall that the 
priority of a late arrival requiring s>x seconds of service 
grows slower than that of the test job. Such a late arrival 
will always be serviced after the test job. On the other hand, 
the priority of a late 5-job with s<x may outgrow that of the 
test job. Assume that a late arrival arrives i seconds after 
the test job. In order to determine how large the service 
requirement 5 of a late arrival may be if it is to be serviced 
before the test job, we consider the case where the priorities 
of the late arrival and the test job are equal at the instant at 
which the test job’s service commences. Equating their 
priorities according to equation (1) we get 

(V-i)/s=V/x 

where V now denotes the delay of the test job. Solving for 

■s. 


s=x(l-i/V), 

yields the maximum service requirement of a job arriving i 
seconds after the test job if it is to be serviced before the 
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test job. Holding V constant, we first obtain the average late 
work conditioned on x and V, i.e., the average cumulative 
service of all jobs which both arrive and depart during the 
test job’s lifetime in the queue. Making use of Poisson ar¬ 
rivals, the average sum of the individual job’s service times 
may be replaced by an integral over the expected service of 
a job arriving at time i (time being measured from the arrival 
of the test job) multiplied by the probability \di of its arrival 
at time i: 


For h(x ) and its derivative we have 

/i(x)=l-X sg{s)ds 
Jo 

+ (X/x)f s 2 g(s)ds, h( 0)=1, (12) 

Jo 

h'(x)=-(X/x 2 ) f s 2 g(s)ds,h'( 0)=0. (13) 

Jo 


W L (x 


>v)=j J 

•^0 J 0 


V rx(l-ilV) 


sg(s)ds\di. 


Substituting y=x(l-i/V), exchanging the order of integra¬ 
tion, and evaluating the inner integral yields 

W L (x,V)=\vf s(l-s/x)g(s)ds. 

Jo 


The average late work can now be obtained by uncondition¬ 
ing this expression with respect to V . With £[ V]= W(x), we 
have 


W L (x)=\W(x)f s(l-s/x)g(s)ds (10) 

Jo 

which is the final expression for the late work. 

The average delay W(x) was defined in equation (2). Sub¬ 
stituting the early work according to equations (3), (4), (6), 
and (9) and the late work from equation (10), we get 


rx r » 

W(x)=W 0 +x| .sW(s)g(.s)d.s-l-Xx W(s)g(s)ds 
Jo Jx 

+ XW(x)f s(l — s/x)g(s)ds. 
Jo 


Solving for W(x) yields the final integro-differential equation 
for the average delay of a job under HRN conditioned on its 
service requirement x: 


Note for later reference that h(°°)=l—p. The second func¬ 
tion is defined as p(x)=xh(x). For this auxiliary function 
and its first two derivatives we have 


p(x)=x—\x sg(s)ds 
Jo 



+ x[ s 2 g(s)ds, 

J 0 

p(0)=0, 

(14) 

p'(x) = l —X f sg(s)ds, 

Jo 

P'( 0)=L 

(15) 

p"(x) = —Xxg(x), 

p"(0)=0. 

(16) 


Note that the functions h(x) and p(x) depend only on X, x, 
and g; they are not functions of the HRN discipline. We 
also define 

V(x)=W(x)h(x)=W(x)p(x)/x (17) 

as a more suitable function for the reduction of the integro- 
differential equation. 

With the definitions of h(x ) and V(x) in equations (12) and 
(17), equation (11) becomes 


rx r 00 

V(x)= W 0 + X j'W(s)g(5)Js+Xx W(s)g(s)ds, 
Jq J x 


and the first two derivatives of this equation are: 


W 0 + xf sW(s)g(s)ds+\X f W(s)g(s)ds 

W(x)= -^-p-^-. (11) 

1 — X J s(l~s/x)g(s)ds 
Jo 

This equation describes the HRN delay function in an 
M/G/l system. It was first derived by Brinch Hansen 1 in a 
slightly different manner based on a geometric argument. 
He then developed an approximate solution for the special 
case of hyperexponentially distributed service times. We 
proceed to generalize his results by deriving the exact 
solution for arbitrary service time distributions. 


DERIVATION OF THE DELAY FUNCTION 

The integro-differential equation for the delay function 
W(x) in equation (11) can be reduced to a homogeneous 
linear differential equation of the second order. We define 
two auxiliary functions to aid in this reduction process. The 
first function, h(x), equals the denominator of equation (11). 


V'(x)=X I W(s)g(s)ds, (18) 

Jx 

V”(x)=-\W(x)g(x). (19) 

Using the identities in equations (16) and (17), equation (19) 
can be rewritten: 

p(x)V"(x)-p"(x)V(x)=0. (20) 

Thus, the problem of solving the integro-differential equa¬ 
tion (11) for W(jc) has been reduced to solving this second- 
order differential equation for V (x). 

We now assume that V(x) has the form 

V(x)=q(x)p(x) (21) 

where q(x) is an unknown function to be determined. Sub¬ 
stituting equation (21) and its second derivative 

V"(x)=q"(x)p(x)+2q' (x)p' {x) + q(x)p"(x) 

in equation (20) yields 

q"(x)/q'(x)+2p'(x)/p(x)=0. 
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The integral of this equation is 


substituting the delay function in equation (23) with C= W 0 : 


log q'{x )+2 log p(x)=log C 

where log C is an integration constant. Equivalently, 
q' (x)= C/p 2 (x), and a further integration yields the unknown 
function q(x): 

r oo 

q(x)= C(\/p 2 {y))dy+D (22) 

J X 

where D is a second integration constant. From equations 
(17) and (21) we have W(x)=x< 7 (x), and with this equality 
equation (22) becomes 

r oo 

W(x)=x C(l/p 2 (y))dy+Dx. (23) 

J X 

Equation (23) is the desired solution for the HRN delay 
function, but it remains to determine the integration con¬ 
stants C and D . 

Equation (1) states that the priority of a job with a zero 
service requirement increases at an infinite rate. Such a job 
will be serviced as soon as the job it finds in service departs. 
Thus, we have the boundary condition W(0)= W 0 which we 
use to determine the integration constant C. Note that this 
boundary condition is consistent with our expressions for 
the early work in equations (3), (4), (6), and (9). Evaluating 
equation (23) for x=0 results in an indeterminate expression 
(zero times infinity) which can be resolved by L’Hospital’s 
rule 


W(0)=W 0 =C 


lim 

ar-»0 


(l/p 2 (y))dy 

J X _ 

\/x 


= C 


lim 

X-*0 


-(1 /P 2 (x)) 

— \/x 2 


With p(x)=xh(x) and /i(0)=l from equation (12), the value 
of the integration constant C follows: 


C=W o h 2 (0)=W o . 


The second integration constant, D, may be obtained by 
using the fact that the average unfinished work U in an 
M/G/l system (as seen by a Poisson arrival) is invariant to 
the scheduling scheme (Reference 2, p. 115): 

U=WJ(1- P ). 

Expressing U as the average remaining service time of the 
currently serviced job plus all the work represented by jobs 
in the queue we have 

r oo 

W 0 + sn(s)ds=W 0 /(l-p). 

J o 


Substituting the expression for n(s)ds from equation (5) 
yields 




W 0 +A sW(s)g(s)ds= W 0 /(l-p). 


We now evaluate this invariance for the HRN discipline by 


W 0 +kW 0 



(1 /P 2 {y))dyg{s)ds 


i 


+ XD s 2 g(s)ds=W 0 /(l~p) 


where the term containing D evaluates as A DES 2 or 2 DW 0 . 
After canceling W 0 and exchanging the order of integration 
in the double integral, we get 


1+A I (1 /p 2 (y)) s 2 g(s)dsdy+2D=l/(l-p). 

•'0 •'O 

According to equation (13), the inner integral is equal to 
-h'(y)y 2 /k. After evaluating the outer integral by making 
use of the equalities p(y)=yh(y), h( 0) = 1, and /i(°°) = 
1-p we have 

1 + [1/(1 —p)-l]+2Z>=l/(l-p). 

It follows that D= 0. With the integration constants deter¬ 
mined, the delay function in equation (23) becomes 


W(x)=x[ (W 0 /p 2 (y))dy . (24) 

J X 

Equivalently, after substituting the auxiliary function p by 
the defining expression in equation (14) we have 

r°° w A 

W(*)= x - - dy . (25) 

y-XyJ sg(s)d.s+\J J 2 g(5)c?5 

The delay functions in equations (24) and (25) are two equiv¬ 
alent forms of the exact solution of the integro-differential 
equation (11). They specify the average delay of a job, con¬ 
ditioned on its service requirement x, in an M/G/l system 
under the HRN discipline. 


PROPERTIES OF DELAY FUNCTIONS 


The HRN discipline strikes a balance between the extreme 
scheduling algorithms of FCFS and SJF. It still favors short 
jobs, but the delay of long jobs is much less excessive than 
under SJF. Figure 1 illustrates the typical forms of the delay 
functions of the three scheduling disciplines. The average 
delay under FCFS is, of course, equal to the unfinished 
work U : 


W FCFS (x)=U=Wj(l-p). 

It is also independent of the service requirement x. The 
delay function for SJF was first obtained by Phipps: 4 


W SJF (*)= 


W 0 


1-kf yg(y)dy 
J o 


Note that the bracketed term in the denominator is equal to 
the auxiliary function p'(x) defined in equation (15); it equals 
one minus the load contribution of all s -jobs with s<x. The 
SJF delay function starts with a value of W 0 and a zero 
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service time 


Figure 1—Properties of the delay functions for HRN, SJF, and FCFS 

slope at the origin, but rises quite rapidly to its maximum 
value W(°°)=W 0 /(l~p) 2 . The delay function for HRN cov¬ 
ers the same range, but its form is quite different. 

The upper integration limits of the two integrals in the 
definition of p(x) in equation (14) may be replaced by infinity 
to yield a function which p(x) will asymptotically approach. 
For very large values of x, say x>L, we shall use this asymp¬ 
tote as an approximation for the auxiliary function p(x): 

p(x)=x-kxES+kES z , x>L. 

The value of L will vary with the given distribution g and 
the desired accuracy. With this approximation, the delay 
function in equation (24) can be evaluated analytically, 

W(x)= WJKl — p)(l— p+kES 2 /x)], x>L, (26) 

to provide an asymptote for the delay function. This asymp¬ 
tote is depicted in Figure 1. Note that the value of W(°°)= 
W 0 /(l-p) 2 can be obtained from this asymptote. The 
slope of the delay function at the origin may be used to 
graphically determine the overall average delay W. By def¬ 
inition, 

W= [ W(s)g(s)ds, (27) 

J o 

and according to equation (18) this expression is equal to 
V'(0)/A. From the definition of V(x) in equation (17) and 
with h( 0)=1 and h'( 0)=0 from equations (12) and (13) we 
get V' (0)= W' (0) and therefore 

W=W'(0)/A. (28) 

Figure 1 illustrates how this equality may be used to obtain 
the overall average delay W from a plot of the HRN delay 
function. 

So far, we have obtained elementary expressions for W(0), 
W(o°), and the asymptote for large service times in equation 
(26). For intermediate service time values of the delay func¬ 
tion, the integral in equation (24) can be evaluated analyti¬ 
cally for simple service time distributions only. In general, 
we will depend on numerical methods for their evaluation. 
In order to deal with the upper (infinite) integration limit, 


we split the integration interval into two partitions, [ x,L) 
and [£,<»). The integral over the latter can then be obtained 
analytically by making use of the approximation in equation 
(26). For the integral over [ x,L ) we use the trapezoidal 
integration technique with subinterval size d such that 
x-md and L=nd. Thus, equation (24) becomes 

[Fu* + F«7TT)5j] d/2+ * w(i)/i (29) 

where W(L ) is given by equation (26). Since 1/p 2 decreases 
monotonically, roundoff errors are reduced by summing in 
decreasing order of/; this corresponds to an integration from 
L to the left. When many values of the delay function are 
needed, an iterative method may be used to increase the 
efficiency of the computation. This iterative method is based 
on the relation 

W{md)=W{{m+\)d)~-^ (30) 

Wp Wp 1 

p z (md) p z ((m+\)d)\ 

which follows from equation (29). The initial value 
W(L)=W(nd) is again given by equation (26), and the iter¬ 
ation proceeds from m=n -1 to m= 1. For m=0, we already 
obtained W(0)=W 0 analytically. 

The overall average delay W is an important performance 
measure. Analogous to the delay function, we use a com¬ 
bination of analytical and numerical methods for its evalu¬ 
ation. Starting from equation (27), we again divide the in¬ 
tegration interval into two partitions: [0,L) and [T, 00 ). The 
contribution of the integral over the interval [L, 20 ) can be 
approximated analytically by making use of equation (26). 
Choosing the simple trapezoidal integration method for the 
interval [0,L), we get 

w=2 [w(jd)g(jd) 

+°W(( j+\)d)g(( j+l)d)]d/2+ W(L)G C (L) (31) 

where L=nd and G c is the complementary cumulative dis¬ 
tribution function, 

G c (x)=l-G(x)=[ g(s)ds. (32) 

Jx 

with G{x) denoting the cumulative distribution function of 
the service time S. Equation (28) shows the relation between 
the overall average delay W and the slope of the delay 
function at the origin. Thus, in addition to its importance as 
a performance measure, the value of W also aids in verifying 
the accuracy of the numerical integration underlying equa¬ 
tion (30). 

THE HRN DELAY FUNCTIONS OF SPECIFIC 
SYSTEMS 

Our result for the HRN delay function in equations (24) 
or (25) is valid for M/G/l systems. To illustrate the behavior 
of the HRN discipline for a representative set of specific 
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distributions, we present four examples with widely varying 
coefficients of variation C v , where C v is defined as 

C V =^ES 2 ~(ES) 2 /ES. 

In the M/D/1 system, service times are constant and C v = 0. 
Erlangian (hypoexponential, Gamma) distributions are char¬ 
acterized by Ck< 1; we shall deal with a 2-stage distribution 
or an M/E 2 /l system. With the exponential distribution in an 
M/M/1 system we cover the case C v = 1. Finally, the 2-stage 
hyperexponential distribution of the M/H 2 /l system will rep¬ 
resent distributions with C v > 1. 

The program for the evaluation of delay functions and 
overall delays was written in SIMULA on a PD P-10. We 
reduced the subinterval size d until we obtained an accuracy 
of at least 3 digits; z/ = 1/100 was sufficient in all cases. For 
a given distribution and load, the delay function was eval¬ 
uated at multiples of the subinterval size according to equa¬ 
tion (30), and these values were subsequently used to obtain 
the overall average delay as indicated by equation (31). The 
SIMULA EXP-function on the PDP-10 has a minimum 
(most negative) permissible parameter value which is around 
-89 [EXP(-89)=2.23 10~ 39 ]. We used this constraint to de¬ 
termine the value ofL for the various distributions. We shall 
list these values of L for the Erlangian, exponential, and 
hyperexponential distributions. For constant service times, 
the delay function can be obtained analytically, thus avoid¬ 
ing the use of numerical methods. 


The MIDI1 system 


For constant service times, the distribution function is 
given by 


g(x)=8(ES) 


where 8 is Dirac’s delta, function. For this distribution func¬ 
tion, the auxiliary function defined in equation (14) evaluates 
as 


\y, y<ES; 

p(y)=\ 

[y-py+pES , y>ES. 

Due to the simple form of p{y), the delay function can be 
obtained analytically. Evaluating equation (24) for this dis¬ 
tribution, we get 


W(x)= 


W 0 (l—p+\x)/(l—p), 
W 0 x/[(1- p)(x- px+ pES )], 


x<ES; 

x>ES. 


Of course, the probability of any job requiring more or less 
than exactly ES seconds of service is equal to zero. The 
average delay of a job is therefore 


W(ES)=W=W 0 /(l-p). 


Note that this expression is equal to W'(0)/\. As one should 
expect in an M/D/1 system, the average delay under HRN 
is equal to the delays under FCFS or SJF. 


The M/EJ1 system 

The distribution function of the 2-stage Erlangian distri¬ 
bution is given by 

g(x)=(2p) 2 x exp(-2px) 

where p.= l/ES. The second moment is ES 2 = 3/(2/x 2 ). For 
the complementary distribution function we have 

G c (x)=l-(l+2/u,x) exp(-2 p,x). 

Evaluating equation (14) for this distribution, we get 

P(y)= y(l-p)+3p/(2p)-py(py+2+3/(2py)) exp (~2p,y). 

For the numerical evaluation of the delay function with 
ES= 1, we set L= 44. Figure 2 depicts the HRN delay func¬ 
tion together with those of FCFS and SJF for a load of 75 
percent. The average overall delays for FCFS, HRN, and 
SJF are 2.25, 1.74, and 1.37 respectively. 



delay 



service time 

Figure 2—Delay functions for hypoexponentially distributed service times 
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The MIM/1 system 

For the exponential distribution function, 

g(x)=/jLexp(-fjLx), G c (x)=exp(-px), 

the first two moments are given by ES= 1/pand ES 2 =2/p 2 . 
The auxiliary function in equation (14) specializes to 

p(y)=y(l-2p)+p(y+2//i)(l-exp(-/Ay)). 

Choosing ES= 1, we set L =89 for the evaluation of the delay 
function based on equation (30). For a load of 75 percent, 
the delay function of HRN is plotted in Figure 3. For com¬ 
parison, the delay functions of FCFS and SJF are also illus¬ 
trated. The average overall delays for FCFS, HRN, and SJF 
are 3.00, 1.98, and 1.56 respectively. 



delay 



service time 

Figure 3—Delay functions for exponentially distributed service times 


The MIHJ1 system 

The 2-stage hyperexponential distribution function is 
given by 

g(x)= abfx exp (- bpx)+ (1 - a) eg, exp (- cjjlx) 

where p=l/ES, a is an arbitrary probability, and b and c 
are positive parameters satisfying a/b+(\-a)/c =\. The 
second moment is ES 2 =2{a/(bp) 2 +(\-a)/(cp) 2 ]. The com¬ 
plementary distribution function may be obtained from 
equation (32): 

G c (x)= a exp (- bp.x)+ (1 - a) exp (- cpx ). 

The auxiliary function p(y) to be used in the numerical 
integration specializes to 

p{y)= y-apy[2-(1+2/(bfxy))(l-exp{-bfxy))]/b 

— (1—a)py[2— (l+2/(cpy))(l—exp(—cpy))]/c. 




service time 

Figure 4 —Delay functions for hyperexponentially distributed service times 
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For the plots in Figure 4 we used the parameters p,= 1 ,a=Vi, 
b- 5, c= 5 / 9 , and p=%; L was set to 17. Again, the delay 
functions of FCFS and SJF are also plotted. The average 
overall delay of HRN is 2.53 as opposed to 4.92 for FCFS 
and 2.06 for SJF. 

In his paper defining the HRN scheduling scheme, 1 Brinch 
Hansen also developed an appealingly simple approximation 
for the delay in an M/H 2 /l system: 



W 0 +xp 2 /(2(l-p)), 

W 0 /[(l-p)(l-p+2W 0 /x)], 


x^2W 0 /p; 


x>2 W 0 /p. 


This approximation was plotted for a load of p=0.71 and the 
parameters a=0.11, 6=0.21, and c= 1.88 which roughly de¬ 
scribe a service time distribution which was observed at the 
University of Michigan. 5 For comparison, we plot the exact 
solution for this distribution and load in Figure 5 (we set 
£5=1 which determined L=47). Again, we also plot the 
delays for FCFS and SJF. The average overall delay for 



service time 

Figure 5—Delay functions for a service time distribution observed at the 
University of Michigan 


overall 

delay 



system load 

Figure 6—Load-dependency of the overall delay under HRN, SJF, and 
FCFS 


HRN is 3.34 compared to 5.80 for FCFS and 2.84 for SJF. 
Figure 5 demonstrates the usefulness of the approximate 
HRN delay function as a rough design tool. Note however, 
that this function approximates SJF more closely than HRN 
for service times of up to several times their mean value of 
one unit of time. This explains why the approximation for 
the overall average delay W as derived in Reference 1, 

W— W 0 +p 2 /[2(jl{ 1— p)], (33) 

evaluates to 2.82, a value which is actually smaller than the 
overall average delay of 2.84 in the SJF discipline. Figure 
6 depicts overall average delays for the University of Mich¬ 
igan distribution as a function of the system load. The bro¬ 
ken curve corresponds to equation (33). The solid curves 
illustrate the load dependencies of the FCFS, the HRN, and 
the SJF disciplines. While the average overall delay under 
HRN is necessarily longer than that under SJF, it is still 
considerably shorter than the overall delay under FCFS. In 
addition to the preferential treatment of long jobs under 
HRN as compared to the SJF scheme (which is illustrated 
by their respective delay functions), the load dependency of 
the overall delay is a further indication of the superior char¬ 
acteristics of the HRN discipline in a batch-processing en¬ 
vironment. 



Performance Evaluation of Nonpreemptive Response-Ratio Schedulers 


481 


CONCLUSION 

The highest response-ratio next (HRN) discipline was first 
suggested by Brinch Hansen 1 as an attractive alternative to 
the first-come first-serve (FCFS) and shortest-job-first (SJF) 
disciplines on which the most common batch-processing 
algorithms are based. He also formulated an integro-differ- 
ential equation for the delay function and derived an ap¬ 
proximate solution for systems with Poisson job arrivals and 
hyperexponentially distributed service times. 

We redeveloped this integro-differential equation in order 
to both keep this article self-contained and to present our 
slightly different derivation technique. We then derived an 
exact solution for this equation by reducing it to a linear 
differential equation of the second order. This solution, the 
delay function for the HRN scheme in equation (25), is valid 
for Poisson arrivals and arbitrary service time distributions. 
For general service time distributions, the delay function 
cannot be expressed in terms of elementary functions. For 
some simple distributions, however, the integral in the delay 
function yields to an analytic treatment. As an example of 
such a simple distribution for which the delay function can 
be expressed by elementary functions, we presented the 
case of constant service times. In general, a combination of 
numerical and analytical methods must be employed to eval¬ 
uate the delay function. We discussed these hybrid methods 
and plotted the delay functions for hypoexponentially, ex¬ 
ponentially, and hyperexponentially distributed service 
times. Brinch Hansen’s approximation for hyperexponen¬ 
tially distributed service times 1 was compared with our exact 
result and its accuracy was discussed. 

The predominant advantage of the HRN scheme is that it 
resembles the SJF scheme with respect to the preferential 
treatment of short jobs without causing the latter’s excessive 


delays for longer jobs. While the average overall delay under 
HRN is slightly longer than that under SJF, it is still con¬ 
siderably shorter than that under FCFS. Intuitively, these 
characteristics should be expected as a consequence of the 
definition of the HRN scheme, but there have been no quan¬ 
titative measures with the exception of Brinch Hansen’s 
approximate results which are limited to hyperexponential 
distributions. Our analysis provides the designer with such 
a quantitative measure which can readily be used to evaluate 
the performance characteristics of a system under HRN for 
a given (or anticipated) distribution and load. 
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Derivation of equilibrium and time-dependent 
solutions to M/M/°°//N and M/M / 00 queueing 
systems using entropy maximization* 

by JOHN E. SHORE 

Naval Research Laboratory 
Washington, DC 


INTRODUCTION 

Queueing theory has provided the basis for remarkable suc¬ 
cesses in the performance modeling and analysis of com¬ 
puter systems. 6,19,21 Because it is clear that computer sys¬ 
tems do not satisfy assumptions made by the stochastic 
process models that are used, this success has been some¬ 
what puzzling; it appears that queueing theory equations 
have wider applicability than is suggested by their classical 
derivations. Buzen has offered one possible explanation in 
terms of “operational analysis.” 4 In this paper we point out 
that certain results in elementary queueing theory can be 
derived simply and with relatively few assumptions by 
means of entropy maximization. This entropy maximization 
viewpoint provides the basis of another possible explanation 
for the widespread applicability of queueing theory formu¬ 
las. 

Analysis by means of entropy maximization has been of 
interest since Shannon 24 showed, for discrete noiseless sys¬ 
tems, that the best encoding of an information source, in the 
sense of enabling the highest information rate over a fixed 
capacity channel, is the one that maximizes the source en¬ 
tropy. In addition to continuing applications in communi¬ 
cation theory, there has been a growing interest in the use 
of entropy maximization techniques for probabilistic analy¬ 
sis and problem solving in other fields. Much of this work 
has been stimulated by that of E. T. Jaynes. Detailed dis¬ 
cussions concerning the motivation, justification, and valid¬ 
ity of various entropy maximization techniques are available 
elsewhere. 5,9,12-15,17,26,27 There have been applications in sta¬ 
tistical mechanics, 10,11,16 traffic networks, 3,8 reliability esti¬ 
mation, 27 production line decision making, 13,29 system sim¬ 
ulation, 5 statistics, 9,20,22 spectral analysis, 1 image 
reconstruction, 30 and general probabilistic problem 
solving. 12-15,17,28 

After summarizing the entropy maximization technique in 
the second section, we obtain in the third section the maxi¬ 
mum entropy solution for the state probabilities of an ab¬ 
stract, general system. All of our results are obtained by 

* The results in this paper were presented orally at the 1976 IEEE Interna¬ 
tional Symposium on Information Theory. 


specializing this general solution. We make some general 
remarks about applications in the fourth section, and in the 
fifth section we apply the model to M/M/°°//N and M/M/ 00 
queueing systems and show how the classical equilibrium 
distributions result. In the sixth section we discuss how 
maximum entropy techniques can be used in deriving time- 
dependent results. After a preliminary application in which 
we consider a pure birth process, we derive time-dependent 
distributions for the M/M/°°//N and M/M/ 00 queues. Discus¬ 
sion follows in the last section. 


ENTROPY MAXIMIZATION 

Given a set of independent propositions {-Si} that enu¬ 
merate all of the possibilities in some situation or problem, 
and given relevant information that is not expressed directly 
as probabilities, one frequently wishes to convert this infor¬ 
mation into suitable probability assignments p(Si) for each 
of the propositions. Entropy maximization is a means for 
doing so. In the usual case, known information is expressed 
as expectations of suitable functions fi(St) defined on the 
set of propositions. The entropy 

MSiHogpCSi) (i) 

1 

is then maximized subject to the constraints 

2 p(S,)-l (2) 

i 

2 fi(S t )p(S t )=( ft) (/= 1,2 ,...). (3) 

i 

The well-known solution is 

p(S t )=ex p [-/So-2 fi(S t )p ,], (4) 

i 

where /3 0 is a Lagrangian multiplier determined by the nor¬ 
malization constraint (2), and where each /3 t is a Lagrangian 
multiplier determined by a known expectation (fi). If the 
“partition function” 

Z=exp(0 o )=2 exp[~2 fi(Si)Pi] (5) 

i l 
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can be evaluated, then the relationship between each La- 
grangian multiplier Pi and its associated constraint (fi) is 
determined by the equation 


/ f\ = _ d Po _ 1 dZ 
Ul/ dp t zdp t • 


( 6 ) 


For continuous functions, the appropriate generalization 
appears to be the maximization of “cross-entropy” 

H=-J dxp(x) log [p(x)/q(x)] (7) 

where q(x) is some “prior” distribution. Like (1), (7) orig¬ 
inated with Shannon 25 (Shannon’s results for continuous 
channels and for source rates relative to fidelity evaluations 
are based on expectations of (7)). Maximization of (7) and 
related functions is discussed by Good, 9 Kullback, 20 and 
Kashyap. 17 The term cross-entropy is due to Good. 9 Equa¬ 
tion (7) has a discrete analog in cases where a prior distri¬ 
bution is known in addition to the expectations (3) (i.e., 
given no information about the prior, (1) assumes a uniform 
prior). 

As an example, consider the following problem: It is 
known that a series of events occurs and that the time 
between successive events can range between 0 and It is 
also known that the average time between successive events 
is (t), i.e., that events occur at the average rate k=l/(t) . A 
function p(t) giving the probability density of the time be¬ 
tween events is needed. The result of maximizing entropy 
subject to the constraint ftp(t)dt=(t ) is p(t)=ke~ u . 5,2S 
Thus, when a finite mean is specified, the distribution of 
inter-arrival times that maximizes entropy is exponential. 
From the maximum-entropy viewpoint, the term “exponen¬ 
tial arrivals” describes processes about which one knows 
only the average arrival rate. If it is further required that the 
inter-arrival times are independent, then we are assured of 
a Poisson process. Derivation by maximum entropy of the 
full exponential family is discussed by Noonan, et al. 22 and, 
in a somewhat different form, by Kullback. 20 


MAXIMUM ENTROPY SOLUTION FOR A MODEL 

SYSTEM 

In this section, we will obtain the maximum entropy so¬ 
lution for the state probabilities of an abstract system. All 
of our later results will be obtained by specializing this 
general solution. The results presented in this section are 
similar to those in Reference 14. We consider a system 
composed of N components, each of which can be in one of 
two “microstates.” A system state S t is defined as an enu¬ 
meration of which components are in which microstates. 
Hence, there are 2 N such system states S t . If one of the two 
microstates is specified as the “microstate of interest” (it 
doesn’t matter which one), then we may define the function 
n(Si) as 

rt(5j)=the number of components that are in the mi¬ 
crostate of interest, given that the system is in 
the system state Si(Q^n(S f )^N). 

Let p(Si ) be the probability that the system is in the state 


5/. We shall now assume that the expectation of n(St) is 
known, and proceed to maximize the entropy of the distri¬ 
bution p(S t ) given this constraint. Specifically, we shall 
maximize 

2 If 

H=-'Z P (S i )logp(S i ) 

1=1 

subject to the constraints 

2N 

i=2 p(s,) 

i=l 

and 

2 N 

<n) = 2 n(S t )p(S t ). (8) 

i =1 

In this case, the partition function (5) can be evaluated 
exactly as follows: 


. N 

z= 2 g(k)e~^ k 

t=l k =0 



(9) 


In (9), & is a Lagrangian multiplier determined by the con¬ 
straint (8) and g(k) is the number of system states S t such 
that n(Si)=k. The relationship between the multiplier /3i and 
the constraint ( n ) can be expressed explicitly by applying 
(6) to (9). The result is 


-0i= 


< n)/N 
1 ~{n)/N 


( 10 ) 


Using (9) and (10), one can express the maximum entropy 
solution (4) for p(S t ) as follows: 

p(S i )=(l+e- fi ’)- N e-*'* 8 * 

-($H <>» 

In deriving (11), we began by maximizing entropy subject to 
the constraint that the expectation of a function n(5 4 ) is 
known. Equation (11) expresses the result explicitly in terms 
of the function n(S i ) and its known expectation (n). 

In applying the result (11), we shall be particularly inter¬ 
ested in the quantity 

p{k) ^probability that the system is in any state S t such 
that n(Si)=k, 

which is obtained easily from (11): 

p(k)= g(k)(l + e~ 01 )~ N e~ 0lk 


-mn 




( 12 ) 


Thus, given the model system and its constraints, the max- 
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imum entropy distribution for p(lc ) is a binomial. This der¬ 
ivation of the binomial distribution was first obtained, in a 
slightly different form, by Jaynes. 14 

In certain applications, one is interested in the limit 
while (n) remains finite. The result for p(k) in this case is 
obtained by using standard methods (e.g., see Reference 7, 
p. 142) to derive an approximation to (12) given (n)<^N, 
yielding 

(13) 

Another probability of interest is 


outcome of the jth coin toss (l<j<N). Thus, a system state 
represents a record of the results of each toss and the 2 N - 
element set {SJ is a complete enumeration of the results 
obtainable by tossing the coin A times. Since p H is the same 
as pi, the combination of (12) and (14) yields the standard 
result 

APPLICATION TO THE EQUILIBRIUM SOLUTION 
OF M/MM/N AND M/M/» QUEUES 


/^probability that any given system component is in 
the “microstate of interest.” 


Since there are 2 N ~ l system states that meet the requirement 
that the given component be in the “microstate of interest,” 
Pi is the sum of p(Si) over these 2 iV_1 states. The function 
n(Si ) is restricted to the values n(Si)= 1,2, . . . ,N. For a 


particular value n(Si)=k, there are 


(?:,') 


system states 


that contribute to the sum of p(Si). Thus, using (11), we 
obtain 


^(VXWO-W 


(n) 
N ‘ 


(14) 


An equivalent form, in terms of the Lagrangian multiplier 
£i, is 

P/=(l+^ 1 ) -1 - 


GENERAL REMARKS ON APPLICATIONS 

In general, application of the foregoing results to a given 
problem requires two steps: First, one must verify that the 
abstract system on which our results were based is an ac¬ 
curate representation of what is known about the given 
problem. Specifically, one must make sure that the set {5/} 
describes completely the set of possibilities implied by the 
given problem, and that the extent of additional knowledge 
is limited to the expectation ( n ) of the function n(Si). If 
more is known, then our solutions will be incorrect in the 
sense that additional information is available but not consid¬ 
ered, although the solutions may be useful approximations. 14 

The second step is to identify the relationship between 
( n ) (or 0i) and parameters contained in the problem state¬ 
ment so that our results for p(S/) or p(k) can be expressed 
in terms of these relevant parameters. 

As an example, consider the problem of computing the 
probability of obtaining k heads in N flips of a coin whose 
single-toss probability of obtaining a head is p H . In applying 
out model system, we identify the yth system component as 
being in the microstate (heads or tails) that records the 


In the notation of queueing theory (Reference 18, p. viii), 
M/M/°°//N refers to a system of N customers and an “infi¬ 
nite” number of servers (effectively equal to N). Both the 
interarrival times for each customer and the service times 
for each server are exponentially distributed. Customers 
arrive for service at an average rate X per customer and are 
served at the average rate p per server (l/p is the average 
time it takes to complete the service of a customer, after 
which the customer returns to the state “not being served;” 
1/A is the average time it then takes for that customer to 
return to the state “being served”). A server is always 
available to begin service immediately on any customer who 
needs it. Recall that the specification of only X and p is 
equivalent, from the maximum entropy viewpoint, to the 
statement that the interarrival and service time distributions 
are exponential. 

In order to apply our model system, we let each system 
component represent one of the N customers, each of which 
can be in the microstate of interest “being served” or in the 
other microstate “not being served.” Thus, the 2 N element 
set {.S’*} represents all possible states of the M/M/°°//N 
queueing system. The function n(5 { ) is then the number of 
customers being served given that the system is in state S, 
and the expectation ( n ) may be interpreted as a time aver¬ 
age. Conventionally (Reference 18, p. 90), p(k) is defined 
to be the equilibrium limit of a time dependent proba¬ 
bility p(k,t). In this context, the probability p/ is the fraction 
of time that a given customer spends “being served” in a 
system that has reached equilibrium. The relationship be¬ 
tween (n) and the parameters of interest (k,p,N) is obtained 
by noting that the following equality must hold at equilib¬ 
rium: 

PiP=(l-Pi)k. (15) 


Hence, using (14) and (10), we have 

X ( n)/N _ 0l 
p l-(n)/N e ' 


(16) 


Substitution of (16) into (12) yields the following result for 
the probability that k customers are being served: 

This is the classical answer (Reference 18, pp. 107-108). 

For the M/M/°° system, the situation is the same except 
that the limit N->°° is taken and the arrival rate X is a total 
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arrival rate (as opposed to an arrival rate per customer) 
independent of the number of customers in service. In this 
case, the equilibrium condition is 


Solving these equations for d{n)/dti^ 0 would yield time- 
dependent functions for the expectation (n;t). Both (21) and 
(22) have the form 


p(n)=\, (18) 

which, when combined with (13), yields the classical M/M/ 
oo result (Reference 18, p. 101): 


d(n) 

dt 


=a-<p(n), 


which has the general solution 


p(k)=e - w ibML, (19) 

Based on work by Benes, 3 a maximum entropy analysis 
somewhat similar to the foregoing was made by Ferdinand 8 
for the machine servicing problem (M/M/1//N) at equilib¬ 
rium. In that derivation, however, the relationship between 
the Lagrangian multiplier and the parameters of interest was 
obtained by assumption and analogy, rather than by deri¬ 
vation. Specifically, for a system in a state S t with n(Si)=k, 
Ferdinand assumed a state “energy” E(k)=-k log(X// a) 
and was thereby able to eliminate the Lagrangian multiplier 
by analogy with statistical mechanics. In obtaining (17) and 
(19), on the other, we eliminated the Lagrangian multiplier 
by means of (10) and the equilibrium conditions (15) and 
(18). 


TIME DEPENDENT SOLUTIONS 


The (^/-constraint equation (8) can represent knowledge 
of ( n / at any particular time; the maximum entropy solutions 
p(Si ) or p(k) then apply to that time. It was only in the last 
section that we chose a time (t->°°) that corresponded, in the 
queueing systems, to equilibrium values of ( n ). Time-de¬ 
pendent knowledge (n;t) of the constraint (8) can therefore 
be substituted meaningfully into (12) or (13) in order to 
obtain time-dependent probabilities p{k,t). For example, 
consider a pure birth process in which customers arrive at 
a constant, total rate X from an infinite (N-> °°) customer 
population. Once having arrived, a customer stays forever. 
The expected number of customers having arrived therefore 
grows constantly at the rate X: 


d(n) 

~dT~ X ' 


( 20 ) 


By substituting the solution of (20) into Eq. (13), and assum¬ 
ing that no customers have yet arrived at t= 0 , we obtain 

„ N _w (kt) k 

p(k,t)=e M -rr-, 


which is the classical result (Reference 18, p. 60). 

In the last section, we transformed the general solution 
(12) for p{k) into specific solutions by relating (n) or the 
Lagrangian multiplier /3 t to the physical constraints X, p, 
and N. We did so by writing down the equilibrium conditions 
(15) and (18), which we repeat here in more explicit form: 

^r^=-/x(n>+X(/V-<n>)=0 (M/M/°°//N) (21) 

=-p(n)+X=0 (M/M/so) (22) 


( n\t)= {ctl<p){ 1- <?-*')+( rc;0> e~ vt . 

For the M/M/°°//N case (21), one has a—XN and <?=/>i+X 
so that 


(n;t)= (l-e- ik+M )+(n;0)e-“ +M . (23) 

AijU- 

By assuming that (n;0)=0 and substituting (23) into (12), we 
obtain the following time-dependent solution for the M/M/ 
°°//N queue: 

[ X -lAT-fc 

1 -W 1 - e ~ ,x+w,) J ■ 

This result appears not to be well-known among queueing 
theorists, although it can be derived from a well-known 
result in reliability theory (Reference 2, p. 78). 

For the M/M/°° case (22), one has a~X and <p-p, so that 

{rv,t)={\lp){\-e- M )+(n\Q)e~ M . (24) 

By assuming that (rc;0)=0 and substituting Eq. (25) into (13), 
we obtain the following time-dependent solution for the M/ 
Ml 00 queue: 

p(k ' ,)= b. (£) «-«'“)*«p[- £ a-e-“)] • 

This is the classical solution, which can be derived from a 
slightly more general result given by Saaty (Reference 23, 
pp. 99-100). 


DISCUSSION 

In what sense does the maximum entropy distribution for 
our model system relate to observations made on corre¬ 
sponding real systems? Jaynes 14 has shown that the maxi¬ 
mum entropy distribution is related to observed frequencies 
in the following sense: Given the imposed constraints, the 
maximum entropy distribution can be realized experimen¬ 
tally in overwhelmingly more ways than any other distri¬ 
bution. Thus, if the analysis includes all of the experimen¬ 
tally operative constraints, then the maximum entropy 
distribution is overwhelmingly the most likely distribution 
to be observed experimentally. 

In general, one can consider experimental confirmation of 
a maximum entropy distribution as evidence that the oper¬ 
ative physical constraints have been accounted for properly 
in the maximum entropy analysis. Conversely, major dis¬ 
crepancies between an experimentally observed distribution 
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and a corresponding maximum entropy distribution is evi¬ 
dence that important physical constraints have been over¬ 
looked. 14 When a maximum entropy analysis does not in¬ 
clude all of the operative physical constraints, the maximum 
entropy distribution may still be a useful approximation. 
The accuracy of the approximation will depend on the im¬ 
portance of the ignored constraints. 

In the case of the queueing formulas derived in this paper, 
we should expect them to apply when a real system’s states 
correspond to those of the model ({Sj}) and when the sys¬ 
tem’s dynamics are most strongly dependent on the con¬ 
strained mean value (n) determined by the stated equilib¬ 
rium conditions. If these criteria are satisfied, then the 
system need not satisfy other assumptions that may be made 
in classical derivations of the queueing formulas. The max¬ 
imum entropy viewpoint may therefore explain the surpris¬ 
ingly widespread applicability of queueing theory results. 
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INTRODUCTION 

Computer system performance is usually evaluated by its 
degree of congestion. Hence analysis of queueing network 
plays an important role in this area. Since in a computer 
system there are two types of participants—processors and 
jobs—degree of congestion is usually studied from two dif¬ 
ferent aspects accordingly. In addition to system through¬ 
put, processor utilization and queue size, system users 
would particularly be concerned with their own response 
times. Many fruitful results in network queues have been 
presented in previous works, 1 " 4 where queue size distribu¬ 
tions were derived. Based upon these distributions, one can 
easily calculate throughput, utilization and even average 
response time. However, little knowledge about response 
time distribution is available. In this paper, we consider a 
central server type of queueing system and investigate equi¬ 
librium behavior of job’s cycle time under certain simplified 
assumptions. Derivations of equations and theorem proofs 
can be found in References 5-10, and hence are omitted 
here. 

A central server queueing system is a closed network 
consisting of a single central server and k parallel servers. 
A job starts his cycle at the time that he enters the queue of 
the central server (central queue). After having received 
service from the central server, the job will join one of 
queues in front of parallel servers (parallel queues) according 
to a probability distribution. When the job leaves the parallel 
server, he will re-enter the central queue and at this moment 
a cycle has just been completed. Considered in the following 
is a simple central server queueing system serving N iden¬ 
tical jobs. Servers are assumed to have independent expo¬ 
nential service time with FIFO queueing discipline. All par¬ 
allel servers are identical. Without losing generality, it is 
further assumed that at time f=0 a tagged job is about to 
join the central queue and the system is in equilibrium. 

Define 


C=cycle time 

number of jobs in the central server at time t, 
i= 0 , 

number of jobs in the ith parallel server at t, 
i=l, ... , k. 

N(t)=(N 0 (t), . . . , N k (t)), system state at time t 
p m i (t;h)=P[N i (t)=m \ N(0 + )=h] 
q(h)=P[N(0 + )=h] 

f(t,k)= cycle time density with k parallel servers. 
{I/j}=exponential random variables with a rate k 
{V)} exponential random variables with a rate p 

m 

T(m)= £ U, 

3=1 

k 

H={h | £ nt=N, ni>0,V/=0, . . . , k} 

1=0 

k 

H t ={h | £ n t =N, n 0 >\, n*>0, Vi=l, . . ., k] 

i=0 

Given that N(0 + )=n and the system enters state m=(m 0 , 
. . . , m k ) after a random time T(n 0 ), with probability \/k 
the tagged job would join the i parallel queue and then his 
conditional cycle time is defined by 

C— T(n 0 )+ 1 Vj (1) 

i=i 

In order to find the distribution of C, it suffices to evaluate 
q(h) and P m i (t;h)=P[N i (t)= m \ N(0 + )=h]. 


QUEUE SIZE DISTRIBUTION 
By using Gordon and Newell’s formula, 2 

/I \n° fc / i \ n } 

*].(-) n y ,v^ 

where a is read as “is proportional to” it then can be shown 9 


X.=service rate of the central server 
p =service rate of parallel servers 
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that 



or 


where 


q(h)=G 



N-no 


V heH 1 , 


( 2 ) 


ns,(r'(ip 

Note that q(h) = q(m) if n 0 = m 0 . The state space H 1 may 
be reduced to one dimension by considering N 0 (t) only. 
(This will be discussed later). 

To evaluate the conditional probability 

Pm i (t;n)=P[N m =m | N(0 + )=n] 

we look into the arrival process to a parallel server. The 
following lemma are due to the fact of exponential service 
distribution. 


Lemma 1 8,9 

n 

Given that T(n)= ^ Vj=s, where If s are iid exponential 

j=i 

random variables, then the conditional joint distribution of 
partial sums (T(m )| m= 1, . . . , n-l}are uniform over (0,5). 

The above statement asserts that conditioning on 
{N 0 (0 + ) = n and T(n)=s}, the first (n- 1) departures epochs 
from the central server (hence the arrival epochs to parallel 
servers) form a uniform order statistics ranged over (0,s). 
Since each departure randomly selects his own parallel 
server, one would expect that the arrival epochs as seen by 
each parallel server would also be a set of uniform order 
statistics. 

Lemma 2 10 



tion (iii) is subject to a binomial distribution, i.e., 
P[Wi(n,s)=w | N 0 (0 + )= n, r(n)=s] 

= B(\Vi | n—\,k) 



n-wi-l 

, Wj=0, 1, . . . , n- 1, 
(4) 


where Wj(n,s)=number of jobs entered the /th parallel 
server during (O,^). 

By virtue of (2), (3), (4) and Lemma 2, the arrival process 
to each individual parallel server during the random interval 
r(Af o (0 + )), can be characterized. 

Now we investigate transient queueing behavior of par¬ 
allel servers from which the conditional queue size distri¬ 
bution P„j(t,h) is derived. 

Consider a single exponential server. Let A(t) be the 
arrival process and D(t ) be the “potential departure proc¬ 
ess” (departure process when the server is never empty). 
Hence, D{t) is a counting process such that the interval 
between successive events is exactly a service time. For a 
given initial condition the queue size can be characterized 
by A(t ) and D(t)- (see Figure 1). 


Given that in the central server queueing system 

(i) N o (0 + )=n o , 

(ii) T(n 0 ) = s and 

(iii) during the period (0 ,s), Wj(<n 0 ) jobs enter the i par¬ 
allel server, 

then these arrival epochs form a uniform order statistics 
ranged over (0,5). 

The probability distribution of condition (i) is given by 
(2). Due to exponential assumption, T(n) has an Erlanigian 
distribution, i.e., 

F[J(n)<t]=r(t|«,X) 

(3) 


Lemma J 5 * 6 * 9 

Let X(t)—A(t)—D(t) and y(0 = inf 0 =su£< X(u). Then the 
queue size Q{t) is given by 

Q{t)=X{t)+ Max{Q(0), - F(t)} (5) 

Lemma 4 9 

Let . . <S v ^s, and 0^7\:s . . <T w <s are two 

independent sets of uniform order statistics over (0,s). De¬ 
fine Z(t) as a one dimension random walk such that (see 
Figure 2). 

Z( 0)=0, 

Z(5 i + )=Z(5r)-l, 

Z(r i + )=Z(Tr)+l and 


Furthermore, since parallel servers are identical, condi- 


Z(s)=w—v. 
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with an equal chance, the probability 


P[X(y)^-r, some ye(0,s)]= 



By definition y(t)=inf 0 s VS i AT(y), it follows that 

P[y(^)>-r]=/ > [X(y)> — r, for all ye(0,s)] 



( 6 ) 


By using Lemma 3 and equation (6), it is not difficult to 
show the following theorem: 


Then every realization of the imbedded chain defined by 


Z{t) at t=S t or Tj has the same probability 



As far as a parallel server is concerned, D(t ) is a Poisson 
process with a rate /x. It is well-known that the conditional 
potential departure epochs *=L • • • , v— 1} 

are uniform order statistics between 0 and s, given that 
D(s) = v. On the other hand, Lemma 2 showed that given 
(i), (ii) and (iii), the conditional arrival epochs {T t , ... , 
T w } are also uniform order statistics. In view of Lemmas 3 
and 4, we conclude that given {N 0 (0)=n o , T(n 0 ) — s, 
A(s)=w, D(s)=v}, every path defined by X(t) as an imbed¬ 
ded chain for 0has the same probability. 

The reflection principle 7 allows us to verify that the number 
of paths of a random walk A(t) from (0,0) to (y,u) which 
cross or touch the line X(t)=-r is equal to the number of 
paths from (0,-2r) to (y,u) (see Figure 3). Let w (or v) be 
the number of increments (or decrements) in X(t) by time 
t=s. The total number of possible paths is equal to 


/w+l A 

V w ) 


and the number of paths cross or touch the line 


X(t) = — r is then 


( w+v\ 
w+r )' 


Since each path can be realized 


Theorem 7 9 

P[Ni(s-)=mi | N(0 + ) = h, T(n 0 )=s , W' i (n 0 ,5) = w’ i ] 

= r(/u,5,« j -m i +w i ) 

_ 7?(/i,^,n i +w i +l)(>Vi-mi-/x5)>v i ! 

(ju,5 , ) mi+1 (w i -mf)! 

=A(m { | ni,Wi,s,ii ) 

where 

u n 

r{u,n)=e~ u — 
and 

7?(«,n)=i e~^. 

v=n v • 
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CYCLE TIME 


Machine Repairman Model, K=°° 


According to equation (1), the conditional cycle time dis¬ 
tribution is given by 
P[C^t | N(0)=n, T(n 0 )=s, W(n 0 >-s) : =w ; ] 

k ni+wi j 

= 2 2 r(t-s\m i ,fi)-A(m i ,\ niWi,s,/i)-j (7) 

i -1 m ( =* 0 K 

Note that since parallel servers are identical, this equation 
can be simplified. By symmetry the first summation and the 
factor \/k can be deleted. From this equation, together with 
(2), (3), (4), and Theorem 1, we can show the following: 

Theorem 2 10 


If it can be shown that 

m —1 / i \ N—m 


q{m,y 


0 0 


r, m= 1,. • • ,N. 


(N—m)\ 

and B(Wi | «,°°) = 1, iff w f =0, for all /=1,2, . . . 
The cycle time density becomes 

*)W 

(N ~ l (p-lf 




fjue 


(N-l 

jl 

^ 1=0 


( 10 ) 


The cycle time density 

f(f,k)=V~ 1 Y 2 " i\k-l) n ~ m U(k,n,i) 

n-0 m -0 i=0 

(x) ^ r ^Ei+rn)R{Xt,n+\)(^j 
-[ r(fit,m-l)~r{pt,m )] 

where 



k+n-i—2 
N-i-l 


m 


i 


U{k,n,i) = 


1 , \ if A:— 1 

N—n—i—3 + k j , otherwise 
N-n-i-l ) 


The expected cycle time is 


E[C]= 



I 

( N+k- 1\ 

l N ) 

] 

N . 

21 
i- 0 

f'iV+jt-l-A 

l N ~‘ ) 

(t)‘J 


( 8 ) 


(9) 


LIMITING CASES 


where p=X//u,. 


Cyclic Queueing Mode, & = 1 
For k= 1, 

/ 1 \ m_1 / 1 \ N ~ m 

q{m)cc (x) (p) ’ m=1 ” 

B(w | «,1)=1, iff w—n— 1. 

The cycle time density becomes 


p N -X N 


N, 


X N 

X N —/i A 


+ v Z P e M 


(N-l!) 


(N-l!) 


( 11 ) 


DISCUSSION 

In a computer model the central server and parallel serv¬ 
ers are usually interpreted as a CPU and k I/O processors, 
respectively. Therefore, a job’s cycle time is equivalent to 
the interval between his two successive I/O completions. 
When the number of I/O processors is increased, the system 
becomes less congested. If the multiprogramming level, the 
CPU rate and the mean cycle time are fixed, it can be seen 
from Figure 4 that the ratio of variance to squared mean ( V/E) 


As mentioned earlier, the state space 


0.20 r 


k 

Hi=[h | /i 0 -L i=l, . . . ,k, ^ A, =N} 

i= 0 

may be reduced to one dimension. Equation (2) implies that 


P[N o (0 + ) — m]=q(m) 

= 2 ?(«)• 
n*n 0 =m 


Hence, 


q{mY 


(N-m+k- 1\ /l\ m_ 7 1 \ N ~ m , . 
( N-m )(x) [kHj 


,N. 



Figure 5—Cycle time densities for different K 
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Figure 6—VAR[C]/| E\C] | 2 when k= 1 (cyclic queueing model) 


X _ ft C.V. 





Figure 9—Cycle time densities of machine repairman model 


of cycle time is increasing in k. Also, illustrated in Figure 5 
is the fact that degree of skewness of cycle time density is 
reduced if the parameter k is. 

When k= 1, the cycle time behavior would remain the 
same if and are interchanged. It can be shown that 


N+l (l-p* +2 )(l-p*) 

' n (i-p N+i y 


( 12 ) 


where p=\//x. 

This ratio takes its minimum at p=l, and is monotonically 
increasing in p for p>l (hence decreasing in p for p< 1). In 
other words, the cycle time is more regular in a balance 
system (i.e., CPU and I/O have compatible processing 
speeds). However, in a congested system random fluctua¬ 
tion of the cycle time is reduced and its regularity becomes 
insensitive when p changes. As shown by (12), for any 
positive p, V/E-+Q as (See also Figure 6). In fact it 

is easy to show that 

f(t;l) = ke- M if A=p, and 
, as A/-*co 

where 0=min (y,p). 


TABLE I.— V/E Ratio When 7V=5 and £[C']=10 




CPU BOUND 

BALANCED 

I/O BOUND 

k 

X 

0.5004 

0.6 

2 


y 

2 

0.6 

0.5004 


V/E 

0.20 

0.17 

0.20 


y 

1.399 

0.393 

0.305 


V/E 

0.203 

0.28 

0.51 


y 

1.164 

0.321 

0.239 


V/E 

0.206 

0.33 

0.64 


y 

1.002 

0.261 

0.186 


V/E 

0.21 

0.38 

0.76 


y 

0.691 

0.165 

0.107 


V/E 

0.22 

0.47 

0.88' 
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For k> 1, analysis of cycle time is very much complicated. 
We turn to look at numerical results. Considered are three 
cases: CPU bound, balanced and I/O bound. For N =5 and 
£[C]=10, it is found that V/E ratio decreases as CPU be¬ 
comes busy. (See Table I. Note that in cyclic queueing 
model balanced system has the smallest V/E ratio). Figures 
7, 8 and 9 present cycle time densities when k= 1, 3, and 
respectively. When CPU is congested, both the machine 
repairman model and the central server model have less 
skew densities. 
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Network access technology—A perspective* 
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Washington, DC 


INTRODUCTION 

The utility of computer networking is widely accepted, as 
is the complexity of accessing network resources and ser¬ 
vices. This complexity must be reduced to support more 
effective user access to and utilization of these resources as 
discussed in References 1 through 3. Accomplishing this 
reduction across heterogeneous systems is a primary objec¬ 
tive of network access support. Specific capabilities which 
have currently been shown to be feasible include: “macros” 
which permit the user to invoke via a single directive a 
complex of individual system and/or communications sub¬ 
network commands which collectively accomplish a specific 
and common function such as “login;” 4,5 common command 
languages for file manipulation and network job execution 
across a heterogeneous collection of systems; 6 “soft” user 
interfaces providing command completion, spelling correc¬ 
tion, and, potentially, correction of minor syntactic 
errors; 4,7,8 uniform interfaces to similar services existing on 
diverse systems such as bibliographic retrieval services; 9,10 
and the emergent area of “intelligent assistance.” 5 

Currently, there are several ongoing projects providing 
varying degrees and types of network access support for the 
interactive user—the population of interest in this paper. 
Appreciating the functions provided by these efforts and 
determining additional functions necessary to maximize ac¬ 
cess support requires establishment of a basic perspective 
on network access. The global objective of this paper is to 
provide such a perspective. This will be accomplished 
through: (i) identifying related research efforts; (ii) identi¬ 
fying problems inhibiting network access support develop¬ 
ment for the appropriate user categories; and (iii) structuring 
and discussing the major components of such support. One 
of the key components of this structure, expert assistance, 
shields the user from the requirement of learning yet another 
language—that of the access system. Expert assistance elim¬ 
inates the requirement that the user become an ‘expert’ on 


* This work is a contribution of the National Bureau of Standards and is not 
subject to copyright. Partial funding for the preparation of this paper was 
provided by the U.S. Air Force Rome Air Development Center (RADC) 
under Contract No. F 30602-77-F-0068. Certain commercial products are 
identified in this paper in order to adequately specify the procedures being 
described. In no case does such identification imply recommendation or 
endorsement by the National Bureau of Standards, nor does it imply that the 
material identified is necessarily the best available for the purpose. 


the access system in order to obtain simplified access to the 
target system. The sixth section of this paper provides in¬ 
sight into how this goal may be achieved. 

Rand Intelligent Terminal Agent (RITA) 

The Rand Intelligent Terminal Agent (RITA) has initially 
been implemented on a shared minicomputer. This imple¬ 
mentation is based on production systems (sets of condition- 
action rules) to encode complex sets of heuristics for han¬ 
dling interactions with users and with external systems. 5,11,12 
RITA performs complex time-dependent tasks over ex¬ 
tended periods of time with minimal intermediate user input. 
This is accomplished through user agents (processes op¬ 
erating in behalf of the user). Since the productions systems 
can modify themselves, RITA agents can learn in the sense 
of dynamically modifying their behavior. 

Two considerations reflected in the design of RITA are 
that the agents maintain knowledge bases containing heuris¬ 
tic assertions, data reflecting system behavior, and user 
preferences, and that these knowledge bases must be mod¬ 
ifiable by the user. The adaptive behavior of RITA agents 
represents a sophisticated approach to the area of access 
assistance. 

NBS Network Access Machine (NAM) 

The Network Access Machine (NAM) at the National 
Bureau of Standards is implemented on a PDP 11/45 running 
the UNIX operating system and represents another shared 
minicomputer based approach to aiding the network 
user. 1,3,13,14 Such support is provided through three primary 
mechanisms: macros which support expansion of simple 
user-entered commands into the command sequences exe¬ 
cutable on specific networks and host computers connected 
to the network; a response analyzer allowing alternative 
responses (typically the expected response plus error con¬ 
ditions which could occur) during the expansion of a macro; 
and control mechanisms (case statements, if-then- else, 

. . .). Collectively, these mechanisms constitute a command 
language level programming language. 

The NAM design is based on the concept of presenting 
one uniform set of user commands which are executable 
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across network boundaries and across heterogeneous host 
systems. In operation, user commands are first expanded 
into the command sequence appropriate to the system being 
accessed, responses are analyzed to determine if the inter¬ 
action is proceeding as expected, anticipated errors are han¬ 
dled directly, and unanticipated errors are presented to the 
user for handling. These capabilities have proven sufficiently 
powerful to support implementation of a prototype common 
command language for file manipulation and network job 
execution. 6 In addition, a uniform interface to a collection 
of bibliographic data retrieval systems has also been dem¬ 
onstrated. 10 

Service specific access support 

The use of computer networks has stimulated interest in 
providing a homogeneous (virtual) interface to multiple het¬ 
erogeneous resources. One such effort has been pursued by 
Marcus at MIT in the information services community. 9,15 
Access to heterogeneous bibliographic retrieval systems is 
provided in such a way as to make them appear similar. 
There are three distinguishing features of this implementa¬ 
tion: emphasizing a particular application, providing a uni¬ 
form virtual interface to heterogenous systems such that all 
systems appear uniform to the user, and stressing access 
support to the naive user. 

The initial interface, called CONIT (Connector for Net¬ 
worked Information Transfer), is based on the MULTICS 
system at MIT. Support includes a master index and the¬ 
saurus to store the vocabulary of the separate data bases 
along with index term interrelationships. In addition, the 
user is provided with a common bibliographic data structure 
in which data elements are structured and interrelated 
among different data bases. Once the desired retrieval sys¬ 
tem is selected, the user enters common commands which 
are then translated into the specific commands required on 
the selected retrieval system. The approach has been dem¬ 
onstrated using the ARPANET for access to the National 
Library of Medicine MEDLINE service and the MIT Intrex 
retrieval system. 


PROBLEMS IN AUTOMATING SERVICE ACCESS 

The objective of network usage is access to computer 
resources or services. Service, here, refers to programs or 
combinations of programs provided by a computer system: 
for example, assemblers, compilers, data base management, 
report generation, and text processing. 

Since users access services, the fundamental problem of 
network access technology is simplification of service ac¬ 
cess. Moreover, in view of the trend toward interrelated 
services such as that provided by the National Software 
Works program production service, 16-18 the problems of in¬ 
terfacing users to services may require consideration of both 
the interface to a service provided on an individual system 
as well as to a collection of services provided by several 


systems. Consequently, Common Command Languages 
(CCLs) across heterogeneous systems are desirable and 
have been shown to be feasible. 6 

Simplification of service access requires identification of 
service user categories and analysis of the factors impeding 
access simplification. 


User categories 

Users of a computer system can be divided into three 
major categories: endusers, applications programmers who 
construct services for endusers, and systems programmers 
who provide the environment in which applications pro¬ 
grammers function. Note that a given individual may func¬ 
tion in all three categories; the importance of the trichotomy 
is its implicit definition of user objectives. 

The global objective of the enduser is access to services. 
In particular, the nuances of the capabilities provided by a 
computer communication system, the general characteristics 
of the system, the locations actually providing the services 
and the availability of other services not of interest to the 
user are of secondary interest. The user would prefer that 
the computer communication system be transparent so that 
one sees services rather than systems and is only required 
to provide that information necessary to utilize the service. 
Further, the user should be relatively unconcerned with the 
precise structure of system command languages, problems 
caused by spelling errors and other requirements to conform 
to the precise syntactic requirements of the system. 

Applications programmers construct services. Provided 
the performance impact is acceptable, high level access sup¬ 
port of computer and communication components is very 
useful. Such support would reduce the need for explicit 
concern with the vagaries of Job Control Languages, permit 
common naming conventions across systems, ease access 
to systems, and support command language level program¬ 
ming. 

Systems programmers will often find the software support 
provided applications programmers of use. However, the 
detailed, highly system specific knowledge required for the 
construction of ‘systems’ software often precludes adoption 
of a high level view. Although network access support mech¬ 
anisms can be used by systems programmers, performance 
constraints will probably limit their applicability. This ob¬ 
servation is consistent with the evidence provided by pro¬ 
gramming language development which demonstrates that 
for some functions one can only program at the assembly 
language level. 

Looking at the objectives of each user category, it appears 
that endusers constitute the major user group for network 
access support. Applications programmers are likely to find 
such support to be of some help, and systems programmers 
seem unlikely to be directly aided in the construction or 
modification of ‘system level’ functions. Even for this re¬ 
stricted audience, developing the required access support 
proves difficult. 
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Access support difficulties 

Access support would be comparatively simple if all users 
had the same problems with the same services. Unfortu¬ 
nately, user perceptions of the difficulty of service access 
are influenced by: 

• user skill 

• rate of change 

• usage intensity 

• access complexity 


User skill 

Developing effective access support would be difficult 
even if user skills were uniform across services. However, 
skills tend to be service dependent and, for each user, span 
the spectrum from “novice” to “expert.” We hypothesize 
that the desired access support level is inversely correlated 
with the user’s skill and, as a result, there is no single “best” 
access support methodology. 


Rate of change 

Rate of change measures the rapidity with which the user 
changes the collection of accessed services and, thereby, 
the rate at which the user must learn about the new (for the 
user) services being accessed. Although we shall not attempt 
to characterize the precise metric or the difference between 
‘fast’ and ‘slow’ rates, it is evident that the higher the rate, 
the greater the learning burden of the user. Thus, we hy¬ 
pothesize that the utility of network access support, as per¬ 
ceived by the user, is positively correlated with the rate of 
change of service access. 

Usage intensity 

Usage intensity measures the frequency of use of a given 
service. We hypothesize that from the viewpoint of an in¬ 
dividual user, for services of equivalent complexity (meas¬ 
ured in terms of some unspecified metric), the utility of 
network access support is positively correlated with the 
(average) time between invocations of the service. This 
seems intuitively reasonable since the longer the interval 
between service accesses, the greater the likelihood that 
required items of information will be forgotten or confused. 

Access complexity 

Service complexity and capability are often positively cor¬ 
related. We hypothesize that the desirability of network 
access support correlates positively with service complexity 
and, thereby, capability. This seems reasonable since more 


complex services typically require more complex interac¬ 
tions for their effective utilization. 


NETWORK ACCESS OBJECTIVES 

Identification of a “best” access support mechanism is a 
reasonable initial objective for network access investiga¬ 
tions. This objective proves unrealistic due to the four fac¬ 
tors identified above which, individually, cannot be opti¬ 
mized due to user differences. Heafner found the same 
difficulty in dealing with user differences for a subset of 
access support functions, namely command languages. He 
concluded that there is no “best” command language. 19 

Since developing a “best” approach is infeasible, we con¬ 
sider an alternative—the determination of access functions 
which are most amenable to automation. Our alternative is 
based on the identification of: (i) service components appro¬ 
priate for network access; (ii) support requirements appro¬ 
priate to these components; and (iii) an integrated set of 
functions corresponding to these support requirements. 

Service access taxonomy 

Access to a service proceeds in four stages: acquisition, 
initialization, utilization and termination. Acquisition en¬ 
compasses those commands required to ‘run’ the service 
together with those required to gain access to the host, if 
required. Similarly, termination encompasses both the proc¬ 
ess of “exiting” the service and logging off the host, if 
appropriate. (It would be inappropriate, or at least ineffi¬ 
cient, to log out if the next service to be accessed is also 
located on the same host.) Initialization covers the setting 
of parameters prior to actual utilization of the service to 
accomplish the user’s objective. 

Given this taxonomy of service access, it follows that the 
objectives of network access support can be characterized 
in terms of the capabilities to be provided for each of these 
stages. 

An assertion 

We assert that service utilization is not amenable to cen¬ 
tralized network level support. Such support can be pro¬ 
vided for the other three stages. 

Justification of this assertion is provided by both organi¬ 
zational and technological considerations. Organizational 
justification follows through consideration of the groups 
which could provide network access support and through 
analogy with the functions of a computing center counselor. 

Network access support of service utilization requires 
explicit knowledge of the service. Since one would not ex¬ 
pect any individual or group of individuals to be knowl¬ 
edgeable in all network accessible services, it follows that 
no single group can support service utilization. Instead, such 
support should be provided either by user groups for the 
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various services, by the service developers (as was done for 
MYCIN, a diagnostic support service experiment for phy¬ 
sicians, 20 and is being done for the ACC AT Command and 
Control testbed, 7,21,22 or through other means as is being 
done in the area of providing homogeneous virtual interfaces 
to heterogeneous bibliographic retrieval services. 9,10,15 In 
this context, it should be noted that the RITA effort, de¬ 
scribed earlier is concerned with the development of 
“smart” user daemons to assist in the utilization of services. 
(Note that although RITA and MYCIN utilize a similar tech¬ 
nology (production systems), RITA support is external to 
the service while MYCIN provides support as part of the 
service.) 

The technological difficulty in supporting service utiliza¬ 
tion reflects its implicit requirement of knowledge of both 
the syntax and semantics of the service. As such, excluding 
trivial cases, it is effectively a specific instance of the arti¬ 
ficial intelligence problem of knowledge representation 
which, in the general case, is unsolved and still a subject of 
study. 

The following discussion establishes the feasibility of sup¬ 
porting service acquisition, initialization and termination at 
the network level. Although the difficulties of providing such 
support for service utilization have been articulated, some 
discussion of this topic is included for completeness. 


Access support objectives 

Network access support should simplify access to new 
(to the user) services and should minimize the need for 
entering relatively unchanged, previously supplied infor¬ 
mation when accessing a previously accessed service. We 
now discuss some of the major requirements implicit in 
supporting these two objectives. 

Accessing new services 

Access support of new services requires: (i) identifying 
the existence of the appropriate service; (ii) providing infor¬ 
mation about the service; and (iii) simplifying the problems 
of initial access. Within this paper we shall not consider 
item (i); however, some consideration has been given to this 
issue as is reflected in the existence of the ARPANET Re¬ 
source Handbook 23 and the objectives of the REX system. 24 

Users require two kinds of information to access a service: 
global descriptive information such as is commonly found 
in manuals, and specific problem solving oriented informa¬ 
tion to assist them in more efficient usage of the service. 
Although either online or hardcopy manuals are appropriate 
for providing global information, a dynamic mechanism pro¬ 
viding highly specific information is required for problem 
solving. Such information would be based on the individual 
user and system. Network level support of dynamic assist¬ 
ance seems feasible for the acquisition, initialization and 
termination stages of service access. Further, it could be 
provided by the service constructor to support more effec¬ 
tive service utilization. 


Dynamic service access assistance requires profiles for 
the acquisition, initialization and termination stages of ser¬ 
vice access. Such profiles are system dependent and user 
dependent because access to services is dependent upon the 
idiosyncracies of the system providing the service and the 
idiosyncracies of the user accessing the service. Their im¬ 
plementation requires resolution of two issues: (i) determi¬ 
nation of the information to be provided, and (ii) develop¬ 
ment of a mechanism for inserting, storing, modifying and 
retrieving profiles. For the three stages of interest, handling 
the service dependency of the profile seems relatively 
straightforward. However, handling user dependencies can 
become quite complex if self-tailoring mechanisms are pro¬ 
vided. 19 

Accessing previously accessed services 

Given that a service has been accessed before, its sub¬ 
sequent access can be simplified through minimizing reentry 
of previously supplied information via: (i) user and system 
profiles automating service initialization, and (ii) expert as¬ 
sistance to eliminate the need for explicit user concern with 
service acquisition, initialization, and termination. Although 
this reduces the need for entering previously supplied infor¬ 
mation, it will probably not eliminate it. Consequently, a 
‘soft user interface’ is desirable to accommodate more flex¬ 
ible input than is presently acceptable. 

The preceding can be summarized in our assertion that 
the key network access support components are: (i) profiles 
for users and services to simplify and automate service ac¬ 
quisition, initialization, utilization and termination; (ii) soft 
user interfaces to eliminate the need for concern with syn¬ 
tactic ‘nits’; (iii) expert assistance to reduce the need for 
entering repetitive information during the acquisition, ini¬ 
tialization and termination stages of accessing previously 
accessed services; and (iv) dynamic tutorial assistance to 
provide specific, pinpointed problem solving information 
online. 

The next four sections summarize existing information on 
the four support components. It should be noted that the 
soft user interface, expert assistance, and dynamic tutorial 
assistance require a knowledge base which is provided in 
profiles. The order in which these components are discussed 
corresponds to a logical progression providing increasing 
support to the user (see Figure 1). However, the boundaries 
between these support functions are not precise and often 
overlap. 


PROFILES 

The word “profile” implies an outline or representation 
of some entity. In access technology a profile is used to 
represent or characterize a user or service. Several groups 
have investigated the area of profiles and the information to 
be maintained therein. 10,19 Since this is an evolving subject 
area, a succinct list of profile contents is lacking. However, 
a minimal list of required entries can be suggested. 
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SOFT EXPERT DYNAMIC 

PROFILES-♦■USER -►ASSISTANCE-►TUTORIAL 

INTERFACE ASSISTANCE 

Figure 1—Spectrum for network access support levels 


The information in a profile parameterizes both the system 
and the user for all stages of service access: acquisition, 
initialization, utilization, and termination. Thus, the profile 
provides the means to free the user from remembering and 
executing the mundane, trivial portions of service access. 
As a specific example, consider the user of multiple sys¬ 
tems—each system requiring different connection protocols, 
passwords, identifications, etc. The profile can be used to 
store the value of these variables for each system and user, 
thus relieving the user from remembering these details. 

The information stored in profiles is of two different types: 
static and dynamic. Static refers to entries that are basically 
unchanging or change very infrequently. The profile entries 
for acquisition, initialization, and termination contain static 
entries. Dynamic refers to information which is typified by 
change. Profile entries associated with service utilization are 
dynamic; for example, although one may use an information 
retrieval system daily, it would be expected that key words 
would not remain the same from session to session. Another 
example of dynamic profile entries is a function of the access 
system design rather than the target system. If an objective 
of the access system is to characterize the user (e.g., ex¬ 
perience level) or the use of a system (e.g., certain se¬ 
quences of commands are habitually repeated), these char¬ 
acterizations are dynamic in nature. 

Static profile entries 

We know that certain static entries are required for both 
the user and system profiles. It would appear that such 
entries are fairly well defined given the access system, target 
systems, and the communications environment. 

User profiles 

In the user profile the following entries are required to 
support the acquisition stage: passwords, user identifica¬ 
tions, and account numbers of the user for all known target 
systems. For the initialization stage, the descriptors required 
for the target system are any default actions to perform 
following login to a target system (e.g., any commands 
which describe special characteristics of a user’s terminal) 
or any default options to issue upon initialization of given 
services on a target system. If the target system is configured 
as part of a computer network, the requirement may exist 
to transfer files from another system to the target system as 
part of the initialization procedure. Any commands which 
should be issued prior to finalizing connection with a target 
system are within the scope of termination. Thus, the user 
may wish to have certain classes of files deleted prior to 
logout. If the target system is configured as part of a com¬ 


puter network, the user may wish for all files (or a subset 
of files) to be transferred to another system in that network 
prior to disconnection. 

System profiles 

In the system profile, there are two subcategories of pro¬ 
file entries associated with the acquisition stage: those rel¬ 
ative to target systems and those relative to the communi¬ 
cations subnetwork (if employed). For the target system, 
the login sequence is required; for the communications sub¬ 
network, the connection sequence to specify the target sys¬ 
tem is required along with any required communications 
control commands. 

The profile entries for the initialization stage reflect any 
requirements of the system prior to utilization: Examples 
are the syntax required to invoke services, and any identi¬ 
fication procedures required prior to accessing given ser¬ 
vices. Profile entries for the termination stage are in two 
subcategories: the target system and the communications 
subnetwork. For the target system, the syntax for logout is 
required, and for the communications subnetwork, com¬ 
mands for closing a network connection are required. 

Dynamic profile entries 

While much is known about requirements for static entries 
in the user and system profiles, determination of appropriate 
dynamic entries is still an open subject for research. Never¬ 
theless, two major generators of such entries can be iden¬ 
tified. Entries corresponding to service utilization are likely 
to fall into the dynamic category. Further, characterizations 
of users and modes of access support utilization maintained 
by the access system require regular updating and, thereby, 
are likely to be dynamic. We conjecture that dynamic profile 
entries will be highly dependent upon specific services being 
accessed and the design of the access system. 


SOFT USER INTERFACE 

The soft user interface refers to those tasks which hu¬ 
manize the system or network interface to a service for the 
user. This section structures the functions to be provided by 
a soft user interface and discusses an approach developed 
by the Stanford Research Institute. 

The functional capabilities provided by a soft user inter¬ 
face can be divided into three major categories: (i) assistance 
in entering individual commands; (ii) assistance in entering 
a collection of commands; and (iii) service identification as 
discussed above. Moreover, each of these categories can be 
subdivided into two major parts according to whether the 
actual command syntax is assumed to be fixed. If this syntax 
is variable, the problem of providing such an interface is 
closely related to the general problem of restricted natural 
language interfaces. 

Assuming the reader knows the appropriate command, 
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assistance in its entry can be provided through command 
completion to minimize the amount of information which 
must be entered, spelling correction, help features to remind 
the user of the command syntax, and user oriented diagnos¬ 
tics to facilitate identification of what went wrong. 

Assistance in reentering collections of commands can be 
achieved through allowing the user to catalog command 
sequences generated in previous interactions with the sys¬ 
tem. This minimizes the amount of information which must 
be reentered. An approach to providing such a mechanism 
is described in the following section. 

The preceding has described means for facilitating the 
entry of commands having rigid syntactic specifications. We 
now discuss how these restrictions might be reduced through 
a more sophisticated interface. 

Restricted natural language interfaces represent a logical 
means for providing syntactically and semantically flexible 
commands. In this context, there is a spectrum of features 
which can be provided. Thus, the interface may query the 
user concerning goals and objectives in an attempt to unravel 
a nebulous user request. Moreover, the interface may use 
a profile system to facilitate self-tailoring in terms of ver¬ 
bosity and user idiosyncracies. As this query facility is ex¬ 
panded it rapidly approaches the capabilities included in 
dynamic tutorial assistance discussed below. 

As an example of the potential power of a restricted nat¬ 
ural language interface, we briefly summarize some of the 
capabilities of a system called LIFER (Language Interface 
Facility with Ellipsis and Recursion), 7,8,21 which has been 
implemented at the Stanford Research Institute. LIFER pro¬ 
vides an automatic facility for handling elliptical (incom¬ 
plete) inputs, a spelling corrector, a grammar editor, and 
language extension through the use of paraphrase. LIFER 
is implemented in INTERLISP and consists of two major 
components. One component is a set of interactive language 
specification functions which are used to define an appli¬ 
cation language which is a supset of a natural language. The 
other component is a parser which interprets natural lan¬ 
guage inputs to translate them into appropriate interactions 
with the application software. 

Production rules are at the core of the LIFER language 
specification functions. The production rules are advertised 
to be easily modified and interactively tested. The intermix¬ 
ing of calls to LIFER for language definition, extension or 


\-/ 

\ OPTIMIZATION FUNCTIONS / 

\-/ 

\ QUERY CAPABILITY / 

\-/ 

\ LEARNING FEATURE / 

\-/ 

\ UNEXPECTED RESPONSES / 

\---/ 

\ FIELD FLAGGING / 

\-/ 

\ TYPESCRIPT / 

\-/ 

Figure 2—Levels of expert assistance 


modification with calls to the parser for utilizing the devel¬ 
oping language system serves as an aid for language devel¬ 
opment. The processing of elliptical inputs is another aid to 
the language builder. This elliptical handling is provided 
automatically by LIFER and frees the system builder from 
incorporating such constructions in the application language. 

For spelling correction LIFER makes use of the spelling 
corrector in INTERLISP. In the event that LIFER does not 
understand an input it prints “user-oriented” error messages 
to indicate what it does understand and suggests correctional 
steps. 


EXPERT ASSISTANCE 

Using an access system often requires that the users learn 
a new command language—the language of the access sys¬ 
tem. Generally, system developers provide a menu of exist¬ 
ing aids; however, in order for the users to tailor those aids 
to individual needs, or to create new aids, the users must 
deal with the language of the access system. This indeed is 
an ironic consequence since it is the goal of access assistance 
to lift users above the idiosyncracies of individual systems. 
Expert assistance refers to the capability of automatic gen¬ 
eration of access system commands to achieve the desired 
user actions. 

Levels of expert assistance 

There are five increasing levels of sophistication for expert 
assistance; however, the underlying commonality is that the 
user informs the expert assistance system of the points at 
which observation of the user’s actions is to begin and end. 
Figure 2 represents the hierarchy of functions which can be 
provided in an expert assistance system. 

The minimum level of access assistance is provided 
through a capability to simply record interactions which 
actually occur between a system and user, and produce the 
access system command sequences required to replay the 
recorded interactions. This level of expert assistance is not 
flexible because it does not handle variations in system 
response or minor (parameter setting) changes in the user 
commands. Greater sophistication is desired. 

The second level of expert assistance permits identifying 
certain fields of the user command or system response as 
variable. Such identification can either be explicitly pro¬ 
vided by the user or by the expert assistance system based 
on its knowledge about commands and services. The iden¬ 
tified fields can then be incorporated as parameters in the 
single command entered by the user to invoke the parame¬ 
terized sequence of commands entered earlier. As an ex¬ 
ample, the expert assistance system might recognize the 
invocation of an editor; given that the syntax for the editor 
is known, the location of the expected file name would also 
be known and could be flagged as a parameter. 

The major drawback of the second level is the frequent 
occurrence of unexpected responses common to a network¬ 
ing environment. The remedy is to incorporate handling of 
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unexpected responses in the assistance system—the third 
level. Two approaches to their handling can be identified: 
(i) a priori identification of all possible responses together 
with incorporation of appropriate actions; and (ii) notifying 
the user when an unexpected response occurs and permitting 
the user to specify the next action to be taken. 

At the fourth level of sophistication, the expert assistance 
system draws upon its own knowledge base of services, and 
uses this information to enhance the generation of command 
sequences. For example, the expert assistance system may 
automatically generate appropriate responses to handle sys¬ 
tem level messages which occur during an interaction and 
should be ignored rather than generating an error. This level 
differs from the third level because the expert assistance 
system dynamically expands its knowledge base rather than 
having given situations identified at design time and pro¬ 
grammed into the system. 

At the fifth level of assistance, the user would be able to 
ask the assistance system to create the specific commands 
by indicating a general class of activity and a target system. 

Currently, expert assistance is viewed as an option avail¬ 
able to the user. There is no intent in the overall design to 
have the expert assistance system suggest to the user that 
certain activities are being repeated and could be incorpo¬ 
rated into a command to the access system; nor is there an 
intent for the expert assistance system to suggest to the user 
that there are more efficient ways to achieve the goals being 
pursued. These capabilities are beyond the current scope of 
expert assistance. However, there is a sixth possible level, 
optimization, which may prove desirable as a future goal of 
such assistance. (Such support mechanisms have been con¬ 
sidered for provision as part of a military message processing 
system. 19 ) 

An implementation approach 

The NBS Expert Assistance System (EAS) is a logical 
extension of the existing Network Access Machine. The 
implementation currently under way will accommodate field 
flagging (by either the user or the EAS if the information is 
contained in its database) and adaptive accommodation to 
unexpected system responses. 

To accomplish these objectives, the EAS initially records 
all characters exchanged between a user and the system as 
demarcated by the user’s overt initiation and termination of 
the recording session. A translator is then invoked to trans¬ 
form these character strings into proper NAM macros which 
may be called and expanded at any time. Variable fields are 
initially accommodated through flagging via a control char¬ 
acter. Subsequently, as the NAM knowledge base is imple¬ 
mented, automatic recognition and flagging of variable fields 
(parameters) will be provided. As identified in Section 6.1, 
there are two approaches to handling unexpected responses. 
In view of the evident impossibility of foreseeing all possible 
system responses, the approach being adopted at NBS is 
the second (user intervention) coupled with preprogramming 
of some ‘expected’ unexpected responses. Thus, the EAS 
will have a minimal learning capability sufficient to permit 


incorporation of unexpected responses when they occur 
and, thereafter, to automatically handle such responses. 

Usage of an expert assistance system by expert users 
constitutes an interesting, potential application. Users who 
are “expert” in the use of specific services on specific 
systems use the expert assistance system to generate the 
appropriate commands in the language of the access system 
to automate access to services for endusers. These “expert” 
users may be from a variety of areas—information retrieval, 
data base management, text processing—and are enabled by 
the assistance system to sit down and quickly create a library 
of commands to support predictable requirements of end- 
users. The “expert” users are shielded from learning the 
command language of the assistance system; they simply sit 
at terminals and enter the commands specific to their ser¬ 
vices. Further, the endusers are shielded from learning the 
command language of the services they wish to access and 
from learning the language of the assistance system. 

A natural enhancement of an expert assistance system 
would be building access aids in the language of the access 
system interactively with the user. Thus, while the system 
is totally responsible for providing aids to support the in¬ 
experienced user, aids might be cooperatively generated 
while interacting with the more experienced user. Such a 
mechanism is discussed in the next section describing a 
highly interactive system in which the user is always being 
guided through the available alternatives. 


DYNAMIC TUTORIAL ASSISTANCE 

A dynamic tutorial system provides for user/service in¬ 
teractions controlled by the access system. The purpose of 
the tutorial system is to guide the user through all possible 
classes of activity offered by an access system. The tutorial 
system continually prompts the user and takes an appropri¬ 
ate course of action dependent upon the user’s action. As 
an example of the use of the tutorial system consider the 
sample interaction below. (The italicized portions are gen¬ 
erated by the user and the other portions are generated by 
the tutorial system.) This example terminates in the invo¬ 
cation of the required access system commands to login to 
the sending system, and transfer a file to the receiving sys¬ 
tem. The profile system contains the default values for the 
sending and receiving systems. If the user elects to use other 
than the default systems, the user may then have the profile 
updated with the new values at the end of the session. 

File Manipulation? yes 

Type of Activity? transfer 
Sending Host is HO ST A? yes 
Receiving Host is HO STB? yes 
Filename? TESTER 

File transfer complete. 

At this point the transfer is completed. This is of course the 
simplest sequence of events. Either the sending or receiving 
system could be other than the default ones; if so, the tu- 
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torial system may have to request user numbers and pass¬ 
words for the new systems if they are not already known. 

The essence of the dynamic tutorial assistance system is 
that the system is in command. It guides the user through 
all activities provided by an access system and executes the 
required commands to achieve the user goals. Additionally, 
the user could have the option to have the tutorial system 
catalog specific interactions so that the user may invoke 
them directly at a later time. Notice that this procedure 
implies a cooperative effort between a dynamic tutorial sys¬ 
tem and an expert assistance system. Using the above ex¬ 
ample, the user could invoke a “transfer” command which 
would result in the transfer of a file from one system to 
another. 


SUMMARY 

This paper has addressed issues relevant to implementing 
access support at the network level. An overview of the 
area of network access was presented, related research ef¬ 
forts were identified, factors which complicate network ac¬ 
cess support were identified, and the major components of 
network access support were structured. A specific imple¬ 
mentation for one of these components, expert assistance, 
was described as it is being constructed at the National 
Bureau of Standards. 

Our development of a structure for network access sup¬ 
port began with the observation that user categories and 
access support difficulties preclude the existence of a single 
general support mechanism. Further, if the objective is pro¬ 
vision of support via a network level group, only the acqui¬ 
sition, initialization and termination stages are reasonable 
candidates. This reflects the reality that service utilization 
requires intricate knowledge of the implementation of a spe¬ 
cific service on a given system; it is unreasonable to expect 
one group (namely, the developers of a network wide access 
mechanism) to possess that level of knowledge for all sys¬ 
tems on one network—much less across network bounda¬ 
ries. 

Although support of service utilization cannot be provided 
at the network level, it is appropriate to provide general 
purpose support mechanisms. To accomplish this, an expert 
assistance system was introduced which can partially auto¬ 
mate tailoring general purpose access mechanisms to meet 
individual objectives. Such a system shields users of access 
support systems from the command languages of those sys¬ 
tems, while enabling them to construct procedures to access 
services on target systems. 
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Adaptive random data generation 
for computer software testing 
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INTRODUCTION 

This paper discusses computer-assisted generation of input 
data for program testing. The test data generation system 
must produce input data that is per specification, unbiased 
by prior analysis of the program to be tested. The test data 
generators evaluated use random generation techniques to 
produce the test data. To reduce the number of sets of test 
data needed to test a program, summary information about 
the performance of previously generated sets of input data 
is used to modify the probability distributions upon which 
the next set of test data is based. Four test data generators 
were evaluated and used to generate test data to exercise 
five testcase programs of various complexity. Observations 
of the results of the actual evaluation runs and of the types 
of structures involved led to the establishment of some 
guidelines for future testing operations. Program verification 
of any sort was not attempted. Manual checks of the normal 
program outputs did detect a number of software errors. No 
attempt was made to automatically isolate or even detect 
software faults. Nor was any comparison with other meth¬ 
ods, manual or automatic, made. 

The Software Testing System developed to use as a tool 
to evaluate the test data generators is similar to others pre¬ 
viously reported. 1-7 These systems require repetition of 
steps to guide and assist test case selection, to execute the 
instrumented program and to analyze testing coverage until 
the test goals have been achieved. The system used here to 
evaluate the test data generators not only provided the sum¬ 
mary reports of testing progress similar to the above sys¬ 
tems, but also provided progress summary information to 
the test data generators which was then used to modify the 
probability distributions controlling the generation of the 
test data. 

Most previous work relating to the generation of test data 
for program testing in some way base the test data on anal¬ 
ysis of the program to be tested. R. L. Sauder 8 determined 
the format and description of the test data by analysis. Other 
systems 9-12 use analysis of the program to guide choice of 
test data when the main goal of testing was usually to exe¬ 
cute all paths through a program. 


Program analysis is subject to any related programming 
errors in the program to be tested which may, in turn, cause 
the generation of test data which is inconsistent with the 
original specification. If any paths are missing in a program 
to be tested, they would not normally be detected through 
use of test data generators which are based on program 
analysis. 

Program test data should be based, as much as possible, 
on the program specifications rather than on the program 
implementation. Sauder’s work was unique in that it re¬ 
quired the user to specify the data relationships thus allow¬ 
ing the “semantics” of the test data to be derived from the 
specification and whatever test plan might exist. The sys¬ 
tems reported by W. H. Burkhart, 13 and K. V. Hanford 14 
are specifically concerned with testing compilers and gen¬ 
erate test data from the formal syntax of the language to be 
processed. These systems are unable to take any semantic 
information, such as relationships or limitations between 
syntactic variables, into account in the generation of test 
data. A new program product recently introduced by Bur¬ 
roughs Corporation 15 does allow specification of both the 
desired syntax and semantics of the desired test data. Each 
of these three systems generate data based on a uniform 
random distribution generator without any concern as to the 
efficiency of such an approach. 

The only reported work involving random test data gen¬ 
eration where techniques were evolved to improve the ef¬ 
ficiency of the generators was in the area of hardware test 
data generators used to test digital logic. 16-17 The original 
concepts have been further extended by K. P. Parker. 18-22 
Parker’s system allows the probability of a particular value 
of any test input to be based on the average or most probable 
value of previously successful sets of test data. 

Although the hardware techniques were developed with¬ 
out regard to software problems, there are many areas in 
common and some of the hardware experience can be ap¬ 
plicable. 23 Although the concept appears valid, hardware 
testing is the process of searching for equipment malfunction 
(assuming a correct design), but software testing is the proc¬ 
ess of somehow verifying the design and implementation of 
the software itself. 
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SOFTWARE TESTING SYSTEM 

The software testing system used for evaluation of the 
test data generators is organized into two phases as shown 
in Figure 1. The first phase, program analysis, inserts mon¬ 
itors in the program to be tested. The monitors allow ob¬ 
servation of flow through the control structure of the pro¬ 
gram, and also observation of changes in the values of 
variables during execution. The second phase, program test 
and evaluation, is the phase which generates test data, runs 
the instrumented program and evaluates the results of the 
tests. In each of the repeated cycles, a new set of input data 
is produced, the program (with monitors) is executed, and 
the results are evaluated. Three of the four test data gen¬ 
erators evaluated were adaptive and utilized monitor-gen¬ 
erated summary information relating to the success of pre¬ 
vious tests in their heuristic algorithms. 

The actual format and values of the input data must be as 
specified for the particular program under test. Special pro- 



Figure 1—Software testing system 


grams supported this function for each of the five test pro¬ 
grams used. In a production environment, a special input 
data specification language would be developed together 
with a processor for that language. Even though each pro¬ 
gram to be tested has special requirements for the form and 
range of its input data, the procedures used for selection of 
the data are generalized and apply regardless of the program 
to be tested. 

Monitoring requirements 

When developing a testing system of any sort, it is im¬ 
portant to know what the test procedure has tested and to 
what extent. This information is important both in deter¬ 
mining what is faulty if a problem is found and in determining 
when sufficient testing has been done. The system devel¬ 
oped monitors both the control structure and range of var¬ 
iables as a means of measuring the ability of the generators 
to “thoroughly” test the software. 

Branch monitoring 

All possible branches within the program graph are mon¬ 
itored. The obvious goal is to attempt to exercise all possible 
branches. Even if all possible branches are exercised, all 
possible paths in the program being tested may not have 
been exercised. In addition, there may be paths which have 
not yet been implemented and which cannot be monitored. 
The simpler measure of branch executions was chosen as 
one measure useful in demonstrating the viability of the 
proposed adaptive random data generators. 


Range of value monitoring 

The user may define a desired range of values to be mon¬ 
itored for each variable of interest. Each range is divided 
into “N” equal size segments (another simple measure cho¬ 
sen as sufficient to demonstrate the viability of the genera¬ 
tors, in this case N=10). Assignments to these variables are 
monitored with the goal of observing data values in each of 
the segments of the range for each variable. The specified 
range of interest is not required to be the same as the ob¬ 
served range of values for any variable. 


Termination conditions 

The need for well-defined termination conditions are par¬ 
ticularly important in a testing system where there may be 
no guarantee that all testing requirements can ever be met. 
The following conditions were adopted, any of which can 
force immediate termination of any testing in progress: 

1. “M” test data sets have been applied to the program 
being tested. (M=50 was used.) 

2. All possible branches have been observed to have ex- 
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ecuted and values have been observed in each segment 
of the range of values for each monitored variable. 

3. The last “P” test data sets were inefficient. In this 
case, if P=10 successive test data sets were not pro¬ 
ductive, termination would occur. A test data set was 
considered not productive if the sum of “new” branch 
executions observed and “new” range of value seg¬ 
ments observed is “R” or less (R=2 during testing). 
“New” branches are those observed which had not 
been executed previously; “new” segments are those 
in which values were observed which previously had 
not been. 

Software test sequence 

The actual procedure of software testing can start after 
the program analysis phase has generated the instrumented 
source program. The system developed (see Figure 2) con¬ 
sists of a central control which, after any necessary initial¬ 
ization, sequences through automatic test data generation, 
execution of the instrumented source program and test eval¬ 
uation. The conditions which cause termination were de¬ 
fined previously. These conditions are checked during the 
test evaluation phase. If termination is not yet required, the 



Figure 2—Software testing system—Program test and evaluation phase 


central control restarts the test generate, execute and eval¬ 
uate sequence. The automatic test data generators are dis¬ 
cussed in detail below. 

Manual verification 

The software testing system developed is not capable of 
automatic verification of the validity of the outputs of a 
program given the automatically generated inputs. The sys¬ 
tem produces summaries of the results of all testing (includ¬ 
ing which branches have not been executed and what range 
of value segments have been observed). This information 
together with the normal program outputs must be manually 
inspected as the final step of program testing. 


AUTOMATIC TEST DATA GENERATION 

The major problem faced in automatic test data generation 
is how to devise adequate tests. Program development pro¬ 
ceeds through a sequence of four steps; specification, de¬ 
sign, implementation, and verification. If the specification 
is assumed correct, then the design and implementation 
should be checked. The test data generation system de¬ 
scribed below generates input data based on the specifica¬ 
tion. It also allows use of manually generated or previously 
saved input data. By monitoring the execution of the pro¬ 
gram, the testing process covers both the design and imple¬ 
mentation. Other systems, which generate test data based 
on analysis of the program being tested only have the ca¬ 
pability of exercising the implementation. Analysis-based 
test data generators produce test data with implementation 
errors reflected in the test data. Therefore, many kinds of 
implementation errors are masked. Specification-based test 
data generation does not have this problem. Since input data 
is based directly on the specification, any errors due to the 
design or implementation procedure are vulnerable to de¬ 
tection. Errors such as missing control paths or incorrect 
handling of input data are likely to be observed using spec¬ 
ification-based test data. 

In a production-oriented software testing system, an input 
data definition language should be available to allow the 
representation of the portion of the program specification 
which defines expected input data in an unambiguous, ma¬ 
chine-recognizable form. In the system implemented, that 
portion of each generator which would normally interpret 
an input data description was specialized for each of the 
programs tested. The specialization was based on the spec¬ 
ification of the input data to be produced and not on analysis 
of the programs tested. 

Recall that the software testing system monitors both the 
sequence of execution and range of value segments. The 
sum of the number of “new” branches and “new” range of 
value segments (where “new” indicates previously unob¬ 
served) is used as a means of comparison of the productivity 
of two test data sets and is used during evaluation of the 
termination condition. 

The judicious selection of a small number of input test 
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data sets from an extremely large domain is another major 
problem when developing an automatic test data generation 
system. The capabilities of four input test data generators 
were evaluated. The four generators are called UNIFORM, 
GAUSS, INVERSE, and NEXT-BEST. All of these gen¬ 
erators, except UNIFORM, are designed to adapt to the 
most effective areas of the input data space. 

UNIFORM test data generator 

The UNIFORM test data generator is the only generator 
evaluated which is not adaptive. Values are chosen ran¬ 
domly over the given legal range of the particular test data 
element being chosen. A uniform probability distribution 
guides the choice so that each value in the range is equally 
likely to be chosen as any other value in the range. In 
addition to allowing direct use of the UNIFORM generator, 
each of the other generators use the UNIFORM generator 
under circumstances explained below. 

GAUSS test data generator 

The development of the GAUSS test data generator was 
motivated by the observation that many program organiza¬ 
tions are not symmetric. The simple example below illus¬ 
trates the asymmetry. The example procedure (Figure 3) 
receives an input parameter X which is an integer in the 
range of 0 through 100. If X is less than 50, a parameter Y 
is incremented. If X is less than 50 and is even, then a 
parameter Z is multiplied by X. If input data to test the 
example procedure is chosen with a uniform distribution, 
one half of the cases, on the average, will test only the first 
conditional. Only one quarter of the cases, on the average, 



Figure 3—Flowchart of example showing control asymmetry 


test both the code which increments Y and which multiplies 
Z by X. If those were major program blocks, it is clear that 
much of the work of a UNIFORM test data generator would 
be ineffective in testing the example program. The asym¬ 
metry also exists in computation of variable values. The 
GAUSS test data generator is designed to bias choice of test 
data by consideration of previously useful test data sets. 

Test data generators are called productive if the data pro¬ 
duced exercise portions of the program not previously 
tested. In terms of the productivity requirement, the pro¬ 
gram organization (or at least that part of interest) is dy¬ 
namic. After each application of a set of inputs, that portion 
of the program which actually was exercised “disappears” 
and is no longer used in evaluation of the performance of 
the generator. The goal of the generators is to produce test 
data which exercises the “remaining (untested) portion” of 
the code. 

Once the generator produces inputs which force execution 
of a branch of the program not previously executed, the 
generator becomes productive. The productivity might.be 
maintained by choosing inputs to again enter the same 
branch, but perhaps test a different subbranch. One can 
usually assume that entry to a particular branch is controlled 
by some subset of the input variables. Thus, if the proper 
input variables are kept “the same” and others are modified, 
it is likely that various subbranches will be exercised. How¬ 
ever, no analysis is performed which would determine the 
subset of inputs which control entry of some branch. If input 
values are chosen randomly under control of a Gaussian 
distribution, and if the standard deviation is small, and if the 
mean represents values needed to enter the branch, the 
generation of values the same as before becomes likely. The 
variation in values should be limited enough to continue 
generation of tests of a productive branch but, at the same 
time, be sufficient to exercise other subbranches. Eventu¬ 
ally, as all the subbranches have been exercised, the pro¬ 
ductivity of test inputs based on that distribution will de¬ 
crease. As this occurs, other successful test sets will be used 
to compute the distribution, and inputs will be produced to 
test other portions of the program. 

Initial GAUSS distribution 

Unless the user provides guidance, the GAUSS test data 
generator has no means of knowing where to set the mean 
and standard deviation initially. In order to provide some 
initial history, the GAUSS generator produces the first “C” 
cases with the UNIFORM generator. For purposes of eval¬ 
uation, “C” was arbitrarily chosen to be four. The UNI¬ 
FORM cases are not generated if any manual or previously 
generated cases are input initially. 

GAUSS generation technique 

The GAUSS generator bases the test data produced on a 
Gaussian probability distribution. Each variable generated 
with GAUSS has a corresponding mean and standard devia¬ 
tion which, together with the defined range of the variable. 
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control the values produced. The mean and standard devia¬ 
tion for each variable are not fixed. These parameters are 
adaptive. Each time the GAUSS generator is activated, the 
mean and standard deviation of each variable is recomputed 
based on the latest “successful” tests. 

It remains to be determined which tests are successful. 
First, a test set is assumed to have certain productivity, 
defined for purposes of evaluation to be the sum of the 
number of “new” branches executed and the number of 
“new” variable value ranges observed. Recall that “new” 
simply means “not previously observed.” The test data 
chosen to compute the means and standard deviations are 
those whose productivity exceeds the average productivity 
of the “M” most recent sets of test data. 

The size of the collection of test data sets considered, 
“M”, also is adaptive. If a number of successive data sets 
are not productive, the range of “M” is doubled up to an 
arbitrary limit. On the other hand, if a number of successive 
data sets are productive, the range of “M” is halved, but 
not less than the original value (in this case, 5). The size of 
the collection of data sets used to define the variable value 
distributions varies because of many of the arguments sum¬ 
marized earlier. A period of high productivity can often be 
maintained by basing distributions on a small number of 
recently successful tests. A period of low productivity is 
often characterized by a search for another productive area. 
A search of this sort is often successful when not biased by 
only a few sets of previously successful data, but by a broad 
representation of all previously successful tests. 

The standard deviations of the distributions used to pro¬ 
duce the test values vary widely. When a particular value 
V 0 of a variable V is useful to force control into a particularly 
productive set of tests, the standard deviation of V may 
become very small as more and more tests with values of V 
near V 0 are produced. If the value of a particular data 
element is not important in that way, the standard deviation 
is likely to be very large, often large enough that the Gaus¬ 
sian distribution over the legal range of values is essentially 
uniform. 

INVERSE test data generator 

The development of the INVERSE test data generator 
was motivated by impatience with the dynamics of the 
GAUSS generator. In particular, because of the procedures 
used to compute the distributions of the various input data 
values, a number of test sets would be produced before the 
deviation was large enough to allow an effective search for 
new productive areas. The problem appeared to be that data 
generated based on current distributions was often non-pro¬ 
ductive and no reliable information was available as to which 
part of the remaining portion of the input space would be 
productive. 

INVERSE and GAUSS commonality 

The resulting INVERSE data generator operates the same 
as the GAUSS generator during computation of variable 


value distributions, establishment of an initial history with 
cases produced by the UNIFORM generator, adjustment of 
the size of the collection of test cases used for distribution 
computations, and random generation of data based on a 
Gaussian distribution. Only after the normal GAUSS pro¬ 
cedures become non-productive is the INVERSE generation 
technique activated. 

INVERSE generation technique 

The only difference between the GAUSS and INVERSE 
generators is how data values are produced after non-pro¬ 
ductive test data sets. If the previous test data set is not 
productive, the INVERSE search procedure is activated 
to attempt to find a new productive region in the input space. 
Successive activations of the search procedure alternate be¬ 
tween standard generation of input data with the GAUSS 
generator and generation of input data with the INVERSE 
procedure. A true inverse distribution is not really appro¬ 
priate. The inverse GAUSSIAN distribution would tend to 
generate probable values at the boundaries of the input data 
space. Rather, it is desirable to produce values not likely to 
be produced with the GAUSS generator, but still provide an 
unbiased search of the rest of the input space. The UNI¬ 
FORM generator can accomplish this easily with the addi¬ 
tion of a simple test of each of the variables generated. If a 
UNIFORM generated value is in the “GAUSSIAN-likely” 
range (MEAN-STANDARD DEVIATION through 
MEAN+STANDARD DEVIATION) where MEAN and 
STANDARD DEVIATION are as normally computed for 
the GAUSS generator, then another replacement value is 
produced (through use of the UNIFORM generator once 
again). When none of the generated values lie within the 
“GAUSSIAN-likely” range, the data values are output and 
applied to the program being tested. Analysis of the pro¬ 
ductivity and recomputation of the variable distributions 
then proceed as previously described with GAUSS. In many 
respects, the INVERSE test data generator can be thought 
of as a random data generator controlled by a UNIFORM 
distribution which has a hole in it, centered at the variable 
MEAN and with radius of the STANDARD DEVIATION. 
It allows a uniformly distributed random search of the area 
not recently productive. 

NEXT-BEST test data generator 

The NEXT-BEST test data generator was motivated by 
the observation that once a particularly productive input 
data set is chosen, other productive data sets can often be 
produced solely by changing only one of the variables in the 
input data set. Calculation of most variable values can be 
considered to be some function of the input variables. Var¬ 
iations of only one input variable will usually affect the 
computed values of some of the program values. Similarly 
the expressions which control branches between basic 
blocks in the program are also functions of the input varia¬ 
bles. The appropriate modification of only one input variable 
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can cause the execution of different basic blocks. The effect 
of modifying only one input variable is essentially an or¬ 
thogonal search of the input data space for other productive 
cases given a known productive case. Any arguments in 
favor of this technique for execution of basic blocks is 
equally valid for testing ranges of variables. 

Initial NEXT-BEST distribution 

The NEXT-BEST test data generator must have at least 
one set of input data values before modification of individ¬ 
ual, randomly-chosen input values can be done. If no manual 
or previously specified data sets are input initially, one input 
data set is produced using the UNIFORM generator. If more 
than one manual or previously specified data set is input, 
the most productive will be used during the next phase of 
the NEXT-BEST procedure, input variable distribution ini- 
itialization. The case chosen will be called the base case. 


Input variable distribution initialization 

The input variables which are randomly selected will not 
be equally productive during the use of the NEXT-BEST 
generator. In an attempt to further improve efficiency, the 
most productive variables are given a somewhat greater 
probability of being used to generate new data sets under 
the implementation of the NEXT-BEST generator. Before 
normal processing begins, each of the input variables is used 
to generate at least one input data set which is a modification 
of the base case chosen after the Initial Distribution phase. 
If the data set generated by modification of a particular input 
variable is productive, then the same variable will be used 
again. If non-productive, the next variable is used. The 
values generated for each of the chosen variables are pro¬ 
duced through use of the UNIFORM generator. A record of 
the total productivity of each variable is kept both during 
this and the final phase. The productivity information is 
utilized during input variable selection. 

NEXT-BEST generation technique 

After Initial Distribution and Input Variable Distribution 
initializations, the major test data set generation phase be¬ 
gins. This phase consists of a number of subsequences of 
test data set generation steps. Each subsequence begins with 
the choice of a base case. The new base case chosen is that 
test data set previously generated which has the largest 
productivity of all those not yet used as a base case. Upon 
completion of a subsequence this base case is marked as 
used. 

After a base case is chosen to start a subsequence, a new 
input variable is chosen. The choice is made randomly but 
with a preference toward previously productive variables. 
To make the choice, an assignment of each variable to a 
portion of the range of real numbers from 0 through 2 is 


made. Let be the size of the portion of the range assigned 
to the zth variable. Assume there are n variables and that Pi 
is that fraction of the total productivity to date contributed 
by the ith variable. Then: 


A random number between 0 and 2 is then chosen under 
control of a uniform distribution. That number will be within 
the portion of the range from 0 through 2 assigned to one of 
the variables and will be used to select that variable. The 
order of the variables across the range does not vary and is 
the same as the order of selection during variable distribu¬ 
tion initialization phase. 

Once a variable is chosen, a new value is selected for it 
to replace its original value in the current base case. The 
selection is under control of the UNIFORM generator. If 
the resulting test data set is productive, the same variable 
is used again for production of the next data set. When a 
data set is not productive, the next variable in the same 
order as chosen during the variable distribution initialization 
phase is selected. By sequencing to other variables in this 
way, variables with poor past productivity can develop a 
more complete history of productivity. The subsequence 
ends when the end of the sequence of variables is reached. 
At that point, the base case is marked as used and a new 
subsequence is initiated. 


EVALUATION AND RESULTS 

The software testing system together with the automatic 
test data generators were evaluated through use of some 
programs chosen as testcases. Limitations of manpower and 
computer time reduced the number of programs chosen to 
five. They were chosen to represent a wide variety of pro¬ 
gram application and complexity. Three of the programs are 
student submissions from an introductory programming 
course and represent table lookup, search techniques, inter¬ 
polation, numerical approximation, divided difference, 
boundary value and game playing applications. In these 
three cases, the general problem was chosen as represent¬ 
ative and one of the programs submitted was arbitrarily 
chosen for each. The other two programs chosen are pro¬ 
duction programs written (and selected) by a staff analyst 
other than the author. These two programs are numerical 
analysis programs concerned with the analysis of stresses in 
turbine blades. The five programs were called: 

Temperature Distribution in a Conducting Solid 
The Series Magnetic Circuit with an Air Gap 
Tictactoe (a tournament) 

Aerospace Analysis 
Turbine Blade Analysis 

(See Lundstrom 24 for a more complete description of each 
of the five testcase programs. 
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Test procedures 

Each of the five testcase programs was “tested” by each 
of the four test data generators. Four of the testcases were 
also “tested” with at least one prespecified set of test data 
values. In order to gather reasonable statistics, each of the 
procedures above was performed ten times. 


Measurement and observation of results 

Control flow and range of value monitors were chosen 
since they reflect, at least to some degree, the thoroughness 
of a collection of test data sets. In the future, other measures 
may prove more useful. The monitors are observed and 
recorded after each test data set is applied to a testcase 
program. This information, when averaged over the ten 
times each generator was used to test each testcase program, 
gives a good feeling for the expected performance of each 
generator. 


Summary of results 

The software test system proved to be useful in detecting 
errors in the programs tested. More importantly, even 
though the input test data generators used cannot be guar¬ 
anteed to find all errors, the generators produce unbiased 
test data which is more likely to thoroughly exercise a pro¬ 
gram, with respect to its specification, than its creator 
would. 

The NEXT-BEST generator, as implemented, never dem¬ 
onstrated productivity better than the others. The results 
might have been different if more initial cases were pro¬ 
duced, if more initial variable productivity history devel¬ 
opment was performed, and if variables known to be non¬ 
productive were used less during the main body of the pro¬ 
cedure. 

The GAUSS generator proved to be most successful in 
situations where a very non-uniform mapping from input 
data space to the program occurs such that only one pro¬ 
ductive region exists. In similar cases with more than one 
productive region, the INVERSE generator was most pro¬ 
ductive, since distribution repositioning can be speeded 
through the UNIFORM “search.” 

The UNIFORM generator represents the type of test data 
generator most used in the past and is most productive when 
the mapping described is uniform. If the mapping is not 
known, then the UNIFORM generator may be slightly more 
productive than the INVERSE generator, most likely due 
to an insufficient “UNIFORM” search for productive areas 
before the main INVERSE procedure begins. Although the 
INVERSE generator is never quite as productive as the 
UNIFORM generator when the mapping is uniform, it is 
sufficiently better when a non-uniform mapping exists (es¬ 
pecially if a good manual case is input) that the INVERSE 


generator can be recommended as the generator to use in 
the many situations when the mapping is not known. 

Recommendations 

An improved generator, called COMPOSITE, has been 
considered, but not evaluated. It should offer better early 
productivity and less sensitivity to the mapping. COMPOS¬ 
ITE would make use of both the UNIFORM and INVERSE 
generators described. However, COMPOSITE would allow 
selection of the generator to be based on past performance. 
The mapping changes for each variable as more and more 
of the program is exercised. By allowing the data generator 
to reflect the current mapping, an increase in productivity 
should be observed. Additional suggested features of COM¬ 
POSITE are described in Lundstrom. 24 

The values of many variables cannot be monitored real¬ 
istically with the fixed-size, fixed-number range of variable 
value divisions utilized in the system implemented. It should 
be possible to specify the number of range segments together 
with the size of each segment (without the constraint that 
each segment be the same size). It is more important that 
the possible values be distributed uniformly across the seg¬ 
ments of the range of values of a variable than to have equal¬ 
sized segments. 


CONCLUSION 

The goals of the work described here were to study random 
test data generation techniques as applied to software test¬ 
ing, to develop and evaluate random data generation algo¬ 
rithms, and to develop a complete software testing system 
to support the study. The software testing system was suc¬ 
cessfully implemented. The test data generators were eval¬ 
uated by using them to test each of five programs chosen as 
testcases; programs which represent a wide variety of ap¬ 
plications. 

The four test data generators developed and evaluated 
were called UNIFORM, GAUSS, INVERSE and NEXT- 
BEST. While each of these was observed to be productive 
in some situations, only INVERSE was consistently pro¬ 
ductive. It was determined that knowledge of the type of 
mapping from the input space to the program could allow 
choice of the appropriate generator for most efficient oper¬ 
ation. Since understanding of the mapping is often difficult 
or time-consuming, and since the system developed did not 
provide tools to support identification of the mapping, the 
INVERSE generator was determined to be the most broadly 
useful over the cases studied. An improved generator, called 
COMPOSITE, was proposed, but not evaluated. 

Although program verification was not attempted, enough 
program faults were manually observed in the test output to 
demonstrate that input test data generation can be success¬ 
fully based on the specification rather than on analysis of 
the program. 
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DP management and administration 


OVERVIEW 

The area of data processing management and administration covers topics 
related to the effective utilization of data processing personnel and technology to 
help achieve the objectives of the organization. Personnel costs continue to grow 
faster than hardware costs, and technology improvements continue to encourage 
more widespread uses of data processing. How our methods of data processing 
management and administration can help us cope with changes such as these is 
theme of the sessions in this area for NCC 78. The sessions can be grouped 
around two general issues. The first issue is the increasing influence of people 
management on the effectiveness of data processing. The sessions related to this 
theme include: time management, motivations, and why is D. P. management so 
difficult? The second issue is the impact of new techniques and technology on DP 
management and administration. Related to this general issue are sessions on: 
information technology and organizational response, security assessment tech¬ 
niques, contribution of planning to information systems productivity, DP auditing, 
and organizational considerations in the allocation of computer resources. 
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Time management for the data processing professional 


by J. F. TOWSEN 

Harrisburg, Pennsylvania 


No longer can the Technical Professional ignore a day-to- 
day interface with peer groups from within the Organization. 

The advent of newer—smaller and faster—operating sys¬ 
tems have made the user a new partner in the development 
of the Corporate Data Processing Strategies. 

But with this new role of partner came a new and different 
type problem for our technical professional. A new problem 
called TIME MANAGEMENT. 

When they started to become more involved in the BIG 
PICTURE, they started just like their peer groups to com¬ 
plain about: 


Meetings 
Interruptions 
Lack of Planning 
Visitors 
Snap Decisions 


Telephone 
Postponed Decisions 
Crisis Management 
No Priorities 
Peer Demands on Time 


The story is always the same, someone or something is 
WASTING my TIME. But have you noticed how most 
people are reluctant to admit their own faults or shortcom¬ 
ings as the real TIME WASTERS. 

Telephones and meetings are very important tools for the 
D. P. Professional. Ideas need to be exchanged, questions 
asked or answered. 

And let’s not overlook the team members of this new 
partnership who can’t COMMUNICATE. Behind most 
every user-D. P. problem, is a breakdown in some form of 
communications. When will they ever realize how much 
time is lost as a result of poor communications, and TIME 
IS MONEY. 

Up to this point I have dwelt on the Data Processor. But 
we must understand that they are no different than other 
managers, most of whom don’t know how to MANAGE 
their OWN TIME. 

How many times have you seen major Data Processing 
projects go down to the wire or even come in late? Why 
does this happen? Why is all of this time and money wasted 
because of unnecessary reruns? 


There are many reasons, but I do suggest that part of the 
problem comes from the D. P. Manager’s inability to manage 
the many demands on their time. The normal management 
functions such as planning, delegation, follow-up and com¬ 
municating get lost in the fast pace of doing business. 

And we can’t forget the cluttered desk and cluttered of¬ 
fice. I believe in the saying “CLUTTERED DESK 
EQUALS CLUTTERED MIND.” I have seen MANY D. 
P. centers and for the most part they are cluttered. 

Along with many other Managers, the D. P. Professional 
must develop some of the basic techniques of good TIME 
MANAGEMENT. 

This seminar will help the manager to develop some of 
these techniques. We will consider some of the following: 


1. Spend more time on FIRE PREVENTION and less on 
FIRE FIGHTING. 

2. Spend 10 or 15 minutes reflecting on the day’s activity 
and planning for tomorrow. 

3. Prepare a THINGS TO DO list detailing the major 
items to be accomplished tomorrow. 

4. Don’t become involved in areas of responsibility di¬ 
rectly assigned to others. 

5. Consider a management by exception program. Only 
the important exceptions reach your desk and require 
your time, energy and ability. 

6. LEARN TO SAY NO. 

7. If your job entails dealing with outsiders, learn to con¬ 
trol these interruptions. Always be in control of the 
situation. 

8. Don’t spend all of your time “getting organized” only 
to find at the end of the day that nothing was accom¬ 
plished. 


And as part of the same seminar, we will look at how to 
get more out of our meetings, and how to “get an extra hour 
out of every day.” 
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Organizational response and 
information technology* 

by RICHARD L. NOLAN 

Richard L. Nolan, David P. Norton, and Company, Inc. 

Lexington, Massachusetts 


INTRODUCTION 

In one instance after another, organizations have remolded 
themselves, not in direct response to great ideas, but in 
response to the development of intervening technology that 
stimulates implementation of those ideas. 1 The remolding, 
however, does not immediately correspond with the advent 
of the new technology. An organizational response process 
is initiated with the advent of the new technology, and then 
evolves over time. The purpose of this paper is to render 
theoretical frameworks that are useful for understanding the 
organizational response process and then to apply the frame¬ 
works to add perspective to contemporary organizational 
developments in respect to information technology. 

RELEVANT THEORY 

There are two theoretical frameworks that are particularly 
relevant to the impact of information technology on orga¬ 
nization. The first is an organizational change framework 
developed by Harold J. Leavitt. The second is an organi¬ 
zational learning framework developed by Richard L. 
Nolan. 

Organizational change framework 

Leavitt’s organizational change framework views the or¬ 
ganization as a complex system consisting of four main 
variables: task variables, structural variables, technological 
variables, and human variables. As shown in Figure 1, 
the variables are interrelated. A change in one results in 
compensatory change in the others. Usually, efforts con¬ 
centrate on structure, people, or technology variables in 
order to effect an improvement in the task variable, but 
other variables also react to the change. 

For example, a structural change such as decentralization 
may have been made with the primary intention of affecting 
the way in which production tasks are carried out. However, 


* The author acknowledges helpful reviews of the ideas and concepts in this 
paper by William E. Bowen, David F. Dantzig, Benjamin S. Porter, Larry G. 
Robbins, and Gonzalo Verdugo, all of Richard L. Nolan, David P. Norton, 
and Company, Inc., Lexington, Massachusetts. 


the change will most likely have an impact on the technol¬ 
ogy, such as a shift toward distributive data processing from 
centralized data processing. The impact on the technology 
variable could be consciously intended, or could be unfore¬ 
seen and often a costly outcome of an intention to change 
only one variable. 

Recognizing that a change in one variable affects the oth¬ 
ers, the remainder of this paper will concentrate on the 
effects of developments in information technology and the 
resultant impact on structure variables or, more specifically, 
organizational response. Information technology is defined 
as any technology that affects the way in which people in 
organizations carry out decision-making and administrative 
activities. The essential concept is that new information 
technology enables implementation of ideas about more ef¬ 
ficient organizational structures. This flow from new infor¬ 
mation technology can be better understood by applying a 
second framework. 


Organizational learning about information technology 

framework 

Nolan’s organizational learning framework views the as¬ 
similation of information technology as an organizational 
learning process involving four growth process variables: 
application portfolio variables, DP (data processing) orga¬ 
nization variables, DP planning and management control 
system variables, and user awareness variables. The appli¬ 
cations portfolio consists of all the automated systems of 
the company. It changes over time in a fairly predictable 
pattern. The other three growth process variables are acted 
on in order to develop an effective applications portfolio. 

In Nolan’s original work he identified four stages of ma¬ 
turity in the evolution of assimilating computer technology. 
The four stages are initiation, contagion, control, and ma¬ 
turity. The assimilation of data base technology requires 
four similar evolutionary stages. Typically, the data base 
initiation stage has coincided with the control stage of com¬ 
puter technology assimilation. The overlapping of the two 
four-stage evolutionary patterns produces the six stage 
evolution of the data resource functions illustrated in 
Figure 2. 
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SOURCE : Adapted from Harold J. Leavitt, "Applied Organizational Change in 
Industry: Structural, Technological and Humanistic Approaches," in 
James G. March, op. cit. , p. 1145. 


Figure 1—Leavitt organizational change framework 


It should be noted that while data base technology is 
typically introduced during Stage III of computer technology 
assimilation, the degree of computer technology assimilation 
is only one of several factors affecting the timing of its 
introduction. Other factors affecting the timing are: (1) avail¬ 
ability and maturity of the technology itself, (2) external 
knowledge of the technology, (3) business needs of the or¬ 
ganization, (4) economic climate of the organization, and (5) 
competitive pressures for data-oriented customer services. 
Companies that are currently in Stage I or early Stage II of 
computer technology assimilation may well initiate data base 
technology in an earlier stage of maturity than has been 
typically observed in the past. 

Nevertheless, the pattern of the data processing budget 
curve is a manifestation of the organizational learning. 
Stages I and II are predominately influenced by the assim¬ 
ilation of computer technology; Stages V and VI are pre¬ 
dominately influenced by the assimilation of data base and 
data communication technologies. Stages III and IV are 
influenced by both types of technologies. 

There are two key concepts that affect the rate and the 


way a company progresses through the six stages: (1) status 
of internal and external bodies of knowledge, and (2) balance 
between control and slack. 

1 . Status of Internal and External Bodies of Knowledge — 
At any point in time, there exists an external or professional 
body of knowledge and an internal, or company, body of 
knowledge on how to effectively manage information tech¬ 
nology. The external body of knowledge is largely codified 
knowledge. It exists in books, seminars and courses. The 
internal body of knowledge is within the company and exists 
in documents and policies, as well as in the minds and 
behavior of the company’s management. The internal body 
of knowledge is largely knowledge that is obtained through 
experience with information technology. It is also obtained 
through books, seminars and courses. But the critical point 
is that experiential knowledge within the company is the 
major factor in progression through the stages. Figure 3 
illustrates the bodies of knowledge concept. The impact of 
external knowledge on the internal body of knowledge helps 
explain why companies that automated their first systems in 
1970 move through the stages differently than companies 
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Figure 2—Six stages of data resource function 


that automated their first systems in 1960. The relative per¬ 
missiveness of an individual company to experiential learn¬ 
ing is also important in understanding how a company moves 
through the stages. 

2. Balance Between Control and Slack —When organiza¬ 
tional learning is viewed as a managed process, two envi¬ 
ronments are balanced—control and slack. 2 In the control 
environment, all financial and performance management 
systems—including planning, budgeting, project manage¬ 
ment, personnel performance reviews and chargeout/cost 
accounting systems—are used to ensure that data processing 
activities are effective and efficient. In the slack environ¬ 
ment, on the other hand, sophisticated controls are notably 
absent. Instead, incentives to use data processing in an 
experimental manner are present. The incentives come in 
the form of what Cyert and March call “organizational 
slack.” 3 When management permits organizational slack in 
the data processing activities, they are committing more 
resources to data processing than are strictly necessary. The 
extra payment achieves another objective—nurturing inno¬ 
vation. When the objective is no longer present or when the 
benefits of overpayment are not apparent, slack is reduced. 
The balance between control and slack is important in ana¬ 
lyzing the functions of each stage of organizational learning. 
Figure 4 shows this balance for the six stages. 

In Stage I, a discretionary expenditure for data processing 


is made, and the objective is designated as the development 
of a particular application. The organization and staffing for 
data processing are focused on the narrow objective. Stage 
I is a period of primary learning about the technology and 
how to use it. It is also a “proving” period to demonstrate 
that the new technology has utility to the organization. Con¬ 
trol is low because a high fixed-cost discretionary expendi¬ 
ture has been made, and there is little reason to monitor 
further low-level expenditures. Slack is low because the first 
applications are narrowly defined replacements for existing 
manual systems. 

Once the first applications reach production status and 
demonstrate the utility of computer applications to the com¬ 
pany, slack is increased to nurture widespread use of the 
computer, marking the transition into Stage II. Systems 
analysts and programmers are often recruited from user 
departments and trained. Formal control is kept low to pro¬ 
mote extensive experimentation with applications in multi¬ 
ple functional areas. Having little experience with computer 
technology, management generally is unaware how fast ap¬ 
plications proliferate. The data processing budget rises rap¬ 
idly during this period because applications often are inef¬ 
ficient and because problems occur when attempts are made 
to integrate operationally-oriented applications with man¬ 
agement’s control and planning requirements. 

The product of inexperienced programmers and virtually 
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no standard for systems development is a portfolio of ap¬ 
plications that are almost impossible to build upon. Thus, 
data processing expenses continue to rise while computer 
services fall off and stagnate. 

When this happens, management initiates actions to in¬ 
crease formal control and reduce slack. The functions of 
Stage III are to develop appropriate management controls 
and to “professionalize” data processing personnel. Often 
management consolidates data processing activities and 
mandates either greatly reduced or zero budget growth. To 
appropriately restructure the applications portfolio, the data 
resource concept 4 is embraced and data-base technology is 
usually introduced to implement the data resource concept. 

This has a confounding impact on the data processing 
budget as illustrated in Figure 2. Despite tight control and 
minimal slack in regard to computer technology, a discre¬ 
tionary expenditure is made for a data base application to 
test the viability of the technology. This is illustrated in 
Figure 4. Once the data processing staff is rebuilt and the 


data base technology is proven, the high control becomes a 
drawback to further progress. Low organizational slack 
impedes the experimentation and innovation needed to re¬ 
structure the applications portfolio under the data resource 
concept. 

In Stage IV, slack again becomes high, but control over 
the data processing department remains high. This is accom¬ 
plished by fundamental reorganization of the data processing 
activities. The responsibility for applications that can be 
associated directly with functional users are assigned to the 
users and programmers and analysts are organized into Ac¬ 
count Teams to assist users. This relieves the data process¬ 
ing department of the no longer logical responsibility for 
operating and maintaining low-level functional systems. The 
responsibility for multifunctional applications is vested in a 
user-represented steering committee. As shown in Figure 4, 
slack concerning data base technology is high, resulting in 
a rapid conversion of the existing applications portfolio to 
a data base structure, as well as a proliferation of integrated 
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Figure 4 —Balance of control and organizational slack through the six 
stages 


applications. The architecture of these applications reflects 
user-orientation through the extensive use of on-line termi¬ 
nals in user departments. 

The low control/high slack for data base technology in 
Stage IV leads to data processing budget growth at rates 
which lead to a second senior management intervention for 
control. Stage V is characterized by data administration and 
a sorting out of data processing department and user de¬ 
partment responsibilities under the data resource concept. 
Slack is reduced and control is increased in order to imple¬ 
ment the standards and policies required. The data base 
structure and specific technology employed are re-evaluated 
in light of current external and internal bodies of knowledge. 
Major restructuring of the data base is typically necessary 
in Stage V. 

Once data resource management is accomplished in Stage 
V, slack can be increased in the specific areas of application 
development where it is required for innovation. The more 
stable and conventional data processing activities then can 


be subjected to a high level of control to ensure maximum 
efficiency. The result is a controlled data processing budget 
growth rate more consistent with the company’s overall 
growth rate. These are the characteristics of Stage VI. They 
indicate organizational acceptance and a relative maturity in 
the Data Resource function. 

Within the context of the two frameworks (i.e., organi¬ 
zational change and organizational learning about informa¬ 
tion technology), the organizational response to contempo¬ 
rary information technology developments can be probed. 

ORGANIZATIONAL RESPONSE AND INFORMATION 

TECHNOLOGY DEVELOPMENTS 

Among the organizational developments in companies’ 
data processing departments that I have observed over the 
past five years or so, three stand out: (1) balancing central¬ 
ization with decentralization, (2) data administration, and 
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(3) distributed processing. Each of these is associated with 
an initiating information technology and stage. Once initi¬ 
ated in a company, the organizational development or re¬ 
sponse to an underlying technology evolves through the 
stages. While there is a “natural” initiating stage, the or¬ 
ganizational development may be initiated in other stages as 
well. Often, earlier initiation precedes a company’s readi¬ 
ness for the organizational development and is dysfunctional 
to effective use and evolution of information technology. I 
call these stages “red flag” stages. 

Organizational response #1: centralization versus 

decentralization 

In the late 1950’s and early 1960’s, the initial high cost of 
computer processing and storage resulted in the centralized 
organization of most data processing departments. Then, as 
shown in Figure 5, the incredible cost/performance improve¬ 
ments in computer processing and storage and development 
of user-oriented programming languages gave rise to consid¬ 
erations for decentralizing parts of the data processing ac¬ 
tivity. Initial considerations were generally given to locating 
systems analysts in user departments followed by consid¬ 
erations to locate computers in the user departments as well. 

The relative absence of planning and control of data proc¬ 
essing activities in Stage II results in a recentralization of 
data processing in Stage III in order to implement the plan¬ 
ning and control structures required to effectively manage 
the activity. Once the controls are implemented, the relative 
centralization/decentralization of the data processing activ¬ 
ity tends to gravitate toward the overall centralization/de¬ 
centralization philosophy of the company. This gravitation 
evolves through Stages IV, V and VI. 

Stages I, II, and III are “red flag” stages because aggres¬ 
sive management action to effect centralization or decen¬ 
tralization which is contra to the natural forces of centrali¬ 
zation in Stage I, decentralization in Stage II, and 
centralization in Stage III can have a disastrous impact on 
the data processing activity. The movement toward central¬ 
ized MIS projects in the 1%0’s when many organizations 
trying such projects were in Stage II is an example of a 
disastrous impact. Many of these companies ended up writ¬ 
ing off their MIS projects as total losses. The organizational 
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Figure 5—Organizational response # 1: Centralization versus 
decentralization 
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Figure 6—Organizational response #2: Data administration 


learning on how to develop applications in multiple depart¬ 
ments and manage large development had simply not de¬ 
veloped sufficiently. 

Organizational response #2: data administration 

Data Administration receives a lot of attention in Stage 
III when data base technology is introduced. However, the 
phenomenon is not understood well enough to write appro¬ 
priate job descriptions and staff it. As shown in Figure 6, 
the activities of data administration are usually incorporated 
into a DP Controller position. During Stage IV the applica¬ 
tions portfolio is restructured using data base technology, 
and a critical mass of data base applications results. Data 
base applications, along with data communications technol¬ 
ogy, enable a user orientation. 

Consequently, the experiential base that evolves by late 
Stage IV creates the need and appropriate organizational 
understanding for establishing data administration in Stage 
V. As the application portfolio continues to evolve with 
higher-level integrated applications, the opportunity for 
gaining competitive advantage through effective manage¬ 
ment and use of data resources emerges. This will lead to 
the position of Data Planner in Stage VI. 

Stages I through IV are “red flag” stages for data admin¬ 
istration because of the lack of critical experiential knowl¬ 
edge. However, Stages III and IV represent important op¬ 
portunistic stages for laying the groundwork for effective 
data administration. 



ORGANIZATIONAL 

RESPONSE 

UNDERLYING 

INFORMATION 

TECHNOLOGY 

INITIATING 

STAGE 

RED 

FLAG 

STAGES 

DEVELOPING 

STAGES 

DISTRIBUTED 

PROCESSING 

• MINI/MICRO¬ 
COMPUTER 

• TERMINALS 

III 

I, II 

IV, V, VI 

VICE PRESIDENT 

OF 

DATA RESOURCES 

IV 

I, II, III 

V, VI 


Figure 7—Organizational response #3: Distributed processing 
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Figure 8—Stage VI data resource organization 


Organizational Response # 3: Distributed Processing 

Distributed Processing is a direct result of technology 
developments in mini/micro computers, terminals, and data 
communications. As shown in Figure 7, distributed proc¬ 
essing is usually initiated in Stage III. If initiated in Stages 
I or II, the technologies are usually overwhelming to the 
data processing department from both a technical standpoint 
and a management standpoint. 

In Stage III the critical control structures are in place so 
as to enable a managed balancing of the relative centrali¬ 
zation/decentralization of the data processing activity to cor¬ 
respond with the overall company philosophy. 

The evolving complexity of distributed data processing 
within the context of an extensive applications portfolio that 
impacts every aspect of the business, leads to the need for 
the participation of a high-level data processing manager on 
the senior management team. In Stage VI, this is reflected 
in an organizational position such as a Vice President of 
Data Resources. Figure 8 illustrates the Stage VI organiza¬ 
tional structure. Parts of the functions are distributed into 
the user departments dependent upon the company’s overall 
organizational philosophy. 


SUMMARY AND CONCLUSION 


Organizations have been and are being remolded by devel¬ 
opments in information technology. The advent of the com¬ 
puter has set off this remolding process, and the process has 
been quickening ever since. 

The first remolding was initiated by breakthroughs in com¬ 
puter processing and storage technologies, and the innova¬ 
tion of higher level languages. The organizational response 
was search for the best balance between centralized and 
decentralized structures. The overall issue was never re¬ 
solved because its resolution depended on the individual 
company’s philosophy about centralization and decentrali¬ 
zation. 

Data base and data communications technologies set off 
another remolding process of organization structure. We are 
just beginning to witness the way that this remolding is 
shaping up. The six stages of the evolution of the data 
resource function is helpful to understanding current devel¬ 
opments as well as future organizational developments. 

Distributed processing has been set off by developments 
in mini-micro computer and terminal technologies. The main 
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effect of these technologies has been added complexity to 
the data resource function. 

In conclusion, three points are important: 

(1) Developments in information technology create orga¬ 
nizational responses. 

(2) Organizational responses of a company are dependent 
upon its data processing stage of maturity. The stage 
is a result of the organizational assimilation of external 
and internal bodies of knowledge. The rate of knowl¬ 
edge assimilation is greatly influenced by the balance 
of organizational slack and control. 

(3) The current information technology and relative stage 
status of most companies has created an extremely 
complex data processing management environment 
that defies simple solutions. Organizational theories 


and frameworks are necessary to sort out the phenom¬ 
ena so that effective management action can be exe¬ 
cuted. 
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Are statistical data bases secure?* 


by DOROTHY E. DENNING 

Purdue University 
West Lafayette, Indiana 


INTRODUCTION 

Statistical databases contain sensitive information about in¬ 
dividuals. Their objective is providing access to summary 
statistics about groups of individuals, while denying access 
to the records pertaining Jo any particular individual. But 
this objective is difficult to meet, as seemingly innocuous 
summaries contain small vestiges of the original data. By 
correlating enough summaries, confidential data may be de¬ 
duced and the privacy of some individual therefore compro¬ 
mised. 

As more and more of these databases are put on-line, the 
problem of preventing their compromise is of growing con¬ 
cern. Several questions have been raised: 

• How easy is it to compromise a statistical data base? 

• Under what conditions are statistical data bases secure? 

• Are these conditions usually met? 

• Is it practical or even possible to impose these condi¬ 
tions on statistical data bases? 

Recent studies reveal that the problem is even more dif¬ 
ficult than at first believed. Methods once thought to signif¬ 
icantly reduce the threat of compromise—e.g., refusing to 
issue summaries about small groups of individuals—are in 
fact easy to circumvent. This paper surveys some of these 
studies. We begin with a general model of a database. 

DATABASE MODEL 
Components 

Consider a statistical database containing records of in¬ 
formation about n individuals. Each record contains cate¬ 
gory and data fields. Categories are used to select subgroups 
of individuals having common characteristics; they need not 
be disjoint from the data values. There may also be a unique 
identifier field, which is neither category nor data. Statistical 
summaries are requested from the database with queries 
which apply to subgroups of individuals. 

Table I shows a database of size n=12 containing infor¬ 
mation on customer accounts for a bank. Each individual 


* This work was supported in part by NSF Grant MCS75-21100. 


belongs to exactly one category in each of these sets: 

Sex: M, F 

Profession: LAWYER, DOCTOR, 

JOURNALIST, 

PRESIDENT, SENATOR, 
BUDGET DIRECTOR 

Board Member: YES, NO 

(Number of) Overdrafts: any integer 2=0 

Each individual also has data values for: 

(Number of) Overdrafts: any integer 2=0 

Amount (of Overdrafts): any integer ^0 

All examples will refer to this database. 

Compromise 

Compromise occurs whenever it is possible to deduce 
from the responses of one or more queries information not 
previously known about an individual. The compromise is 
positive if it reveals that the individual belongs to some 
category or has a particular data value. The compromise is 
negative if it reveals only that the individual does not belong 
to some category or have a particular data value. For ex¬ 
ample, learning that individual L had 50 overdrafts is a 
positive compromise; learning only that he had at least one 
overdraft is a negative compromise. Partial compromise 
occurs when information about at least one individual is 
deduced; complete compromise occurs when everything in 
the database is deduced. A database is strongly secure if 
both positive and negative compromise are impossible; it is 
weakly secure if only positive compromise is impossible. 

Queries 

Researchers have studied two basic forms of queries: 
characteristic-specified and key-specified. Characteristic- 
specified queries request statistics about all individuals in 
the data base satisfying a given characteristic; a character¬ 
istic is a logical formula using categories as operands and 
Boolean operators: and (•), or (+), and not (-). For a char- 
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acteristic C, the set of records satisfying C is called the 
query set X c of C. For example, the characteristic 
C=M(LAWYER+DOCTOR), specifying all male lawyers 
and doctors, has query set X c consisting of the records for 
individuals A, D, E, and H. We shall use relations in the 
specification of characteristics, e.g., OVERDRAFTS=£2, 
since these are simply abbreviations for the “or” of several 
values, e.g., O-OVERDRAFTS + 1-OVERDRAFTS + 2- 
OVERDRAFTS. 

Studies of compromise of statistical data bases have con¬ 
sidered characteristic-specified queries q(C) having one of 
three forms: 

COUNT(C)= | X c |, where | X c | is the size of X c ; 
SUM(C;j)= X v u> where v u is data field j of record i; 

i€X C 

select(C;j)=select Vij, where select is MEDIAN, SMALL- 

leX c 

EST, LARGEST, etc. 

The query COUNT(C) simply returns the number of indi¬ 
viduals satisfying characteristic C. The query SUM(C;j) re¬ 
turns the sum of the values in data field j for all individuals 
satisfying C. Note that the mean of these values can be 
computed from 

ME AN(C ;j)=SUM(C; j)/COUNT(C). 

For select= MEDIAN, the query MEDIAN(C;j) returns the 
median value in data field j for all individuals satisfying C. 
Examples of queries are: 


Formal Query Answer Informal Statement 


COUNT(M LAWYER) 
COUNT(M LAWYER- 
(OVERDRAFTS >10)) 

SUM(L A W YER+ 
DOCTOR; 
OVERDRAFTS) 
SUM(BUDGET 
DIRECTOR; 
AMOUNT) 
MEDIAN(LAWYER+ 
DOCTOR; 
OVERDRAFTS) 


3 number of male lawyers 
2 number of male 

lawyers having more 
than 10 overdrafts 
58 total number of 
overdrafts of all 
lawyers and doctors 
total amount of 
overdrafts of all 
budget directors 
1 median number of 
overdrafts of all 
lawyers and doctors 


$100,000 


Key-specific queries request statistics for a set of k indi¬ 
viduals identified by a list of keys I. The keys are typically 
the names of individuals. For a key set I, the set of records 
identified by the keys is the query set X t of I. The query set 
size, m, is fixed for all queries. Queries for sums and selec¬ 
tion queries are expressed as for characteristic-specified 
queries, with a key set I substituted for a characteristic C; 
e.g., SUM(I;j). Since the size of a query set is always m, 
queries for counts are not applicable. 

Key-specified queries are of less practical interest than 
characteristic-specified queries, since statistical databases 


generally do not give out data about particular individuals. 
However, if each individual is identified by a unique set of 
categories, any key list can be expressed with characteris¬ 
tics. For example, the key list (A, B) can be expressed as 
the characteristic (M LAWYER BOARD MEMBER+M- 
JOURNALIST). Thus, results which show that a database 
can be compromised with keys apply also to characteristics. 
However, caution must be taken in interpreting these re¬ 
sults. Even though keys can be formulated as characteris¬ 
tics, it is not possible to do so without knowledge of the 
characteristics identifying the individuals named. Without this 
information, it is more difficult to control the composition 
of the query sets. For this reason, achieving compromise with 
characteristics may be more difficult than with keys. On the 
other hand, achieving compromise with characteristics may 
be easier than with keys since counts can be obtained for 
variable-size query sets. 


METHODS OF COMPROMISE 

Using small or large query sets 

In one of the first published papers describing the problem 
of securing statistical databases, Hoffman and Miller de¬ 
scribed a simple algorithm for compromising a data base 
responding to COUNT queries. 1 Their algorithm is based on 
the principle of using queries which return small counts to 
isolate an individual. For example, consider these two quer¬ 
ies and responses: 

COUNT(M BUDGET- 
DIRECTOR) =1 

COUNT(MBUDGET- 
DIRECTOR (OVERDRAFTS>0)) = 1 

If it is known that L is a male budget director, then the second 
query reveals that he had at least one overdraft. In general, 
if it is known that an individual belongs to categories 
Ci, . . . , c k and if COUNT(Ci-c 2 *. . . *c k )=1, then the query 
COUNT(ci-c 2 -. . .-Ck Ck+i) reveals whether or not the indi¬ 
vidual also belongs to category c k +i (according to whether 
or not the response is 1 or 0). Indeed, compromise may be 
possible even if the response to the first query is larger than 
1, say 17, if the response to the second query is the same. 

Hoffman and Millers’ result is equally applicable to quer¬ 
ies which return large counts of all but a few individuals in 
the data base. For example, L’s overdrafts can also be 
determined from the queries: 

COUNT(M BUDGET DIRECTOR^ 11 
COUNT(M BUDGET DIRECTOR- 
(OVERDRAFTSX))) =11 

These results have led researchers to consider databases 
which do not respond to queries q(C) when the query set size 
| Xc | falls outside the range [k, n-k] for some k>0, where 
n is the size of the database. It was anticipated that ks:2 would 
significantly reduce the threat of compromise. 
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TABLE I.—Database Containing Information on Customer Accounts for a Bank 



keys 

Name 


categories 


data 

Sex 

Profession 

Board Member 

Overdrafts 

Amount ($) 

1 

A 

M 

LAWYER 

NO 

1 

10 

2 

B 

M 

JOURNALIST 

NO 

0 

0 

3 

C 

M 

PRESIDENT 

NO 

0 

0 

4 

D 

M 

DOCTOR 

NO 

2 

100 

5 

E 

M 

LAWYER 

YES 

30 

50,000 

6 

F 

F 

LAWYER 

NO 

0 

0 

7 

G 

F 

SENATOR 

NO 

3 

50 

8 

H 

M 

LAWYER 

YES 

25 

10,000 

9 

I 

F 

DOCTOR 

NO 

0 

0 

10 

J 

M 

SENATOR 

NO 

0 

0 

11 

K 

F 

JOURNALIST 

NO 

0 

0 

12 

L 

M 

BUDGET DIRECTOR 

YES 

50 

100,000 


Trackers 

Unfortunately, recent studies have shown this not to be 
the case. Schlorer showed that compromise may be possible 
even for large values of k using a “tracker” technique. 2 To 
illustrate his tracker, suppose k=3 for our database; i.e., no 
responses are given to queries which involve fewer than 
three or more than nine individuals. Consider these queries 
and responses, using as a tracker the characteristic 
M-BUDGET DIRECTOR: 


COUNT(M BUDGET DIRECTOR)=7 
COUNT(M BUDGET DIRECTOR 
+M (OVERDRAFTS=0)) =7 

Because the responses to both queries are the same, it can 
be concluded that no male budget director had no overdrafts; 
therefore, L must have had an overdraft. Palme suggested 
a similar technique for queries that compute means. 3 

To construct a Schlorer tracker requires substantial 
knowledge of the characteristics identifying the individuals 
to be compromised. Denning, Denning, and Schwartz 
showed that general trackers may be found even without 
this knowledge. 4 Such trackers may be used to determine 
the answer to any COUNT or SUM query not answerable 
because the query set size is too small or too large. For 
example, if k=3, the characteristic T=LAWYER + 
DOCTOR is a general tracker. Let C=M BUDGET DIREC¬ 
TOR. Whereas the number of L’s overdrafts cannot be de¬ 
termined directly from queries using only C (since | X<, |= 1), 
it can be determined from queries using characteristics com¬ 
bining C with the tracker T as follows. First, the sum of the 
queries COUNT(T) and COUNT(T) can be used to deter¬ 
mine the size n of the database; this gives n=12. Second, it 
can be deduced that L is the only male budget director from 
the queries COUNT(C+T) and COUNT(C+T), since 

COUNT(C)=COUNT(C+T)+COUNT(C+T)-n 
= 7 + 6-12 


Third, the total number of overdrafts X can be determined 
by adding the responses to the queries SUM(T; OVER¬ 
DRAFTS) and SUM(T; OVERDRAFTS), getting X=lll. 
Finally, since L is the only budget director, the number of 
L’s overdrafts can be determined from the queries 
SUM(C+T; OVERDRAFTS) and SUM(C+T; OVER¬ 
DRAFTS), as 

SUM(C; OVERDRAFTS)=SUM(C+T; OVERDRAFTS) 

+SUM(C+T; OVERDRAFTS)-X 
= 108 + 53 - 111 
=50 

We found further that for k^n/4 almost all data bases 
have general trackers (though finding them may not be easy). 
Trackers may also be applicable for larger values of k. 

Adding dummy entries 

Hoffman observed that data bases restricting the size of 
allowable query sets may also be subverted if records can 
be added to the database. 5 If query q(C) is not answerable 
because Xc is too small, individuals with the characteristic 
C are added to the data base; if Xc is too large, individuals 
with the characteristic C are added. This result, together with 
the tracker results, shows that limiting the size of allowable 
query sets is not an effective security measure. 


Overlapping query sets 

Another simple technique for compromising a data base 
is to construct query sets which have a high degree of 
overlap. For example, consider these queries and responses: 

SUM(LAWYER+F SENATOR; AMOUNT)=$60,000 

SUM(LAWYER; AMOUNT) =$60,010 

Since G is the only female senator, the second query set 


1 
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includes all individuals in the first query set except G. Thus, 
the amount of her overdrafts can be determined by subtract¬ 
ing the response of the second query from the first. This 
observation has led researchers to consider the effectiveness 
of methods which restrict the amount of overlap among 
query sets. 


Linear systems 

Dobkin, Jones, and Lipton considered compromising with 
key-specified SUM queries using query sets of size m. 6 They 
showed that, even if no two query sets can overlap by more 
than one record, compromise may be achievable in linear 
time (in m) without prior information, provided the data 
base is sufficiently large (roughly at least m 2 records). The 
method involves solving systems of equations for some par¬ 
ticular data value. For example, suppose m=3 and let Ai 
denote the amount of overdrafts for the i th individual. Con¬ 
sider these 5 queries: 

Q 1 =SUM(A,B,C; AMOUNT) 

Q 2 =SUM(D,E,F; AMOUNT) 

Q 3 =SUM(A,D,G; AMOUNT) 

Q 4 =SUM(B,E,G; AMOUNT) 

Q 5 =SUM(C,F,G; AMOUNT) 

=A 1 +A 2 +A 3 =$ 10 

= A 4 +A 5 +A 6 =$50,100 

=A X +A 4 +A 7 =$ 160 

= A 2 +A 5 +A 7 =$50,050 

= A 3 +A 6 +A 7 =$ 50 

G’s overdraft amount can be determined from 
(Q3+Q 4 +Q 5 -Qi-Q 2 )/3=$50 

Davida et al. 10 and Kam and Ullman 7 considered compro¬ 
mise under similar conditions. 

Schwartz, Denning, and Denning extended these results 
to weighted summing queries. 8 We were surprised to learn 
that if the weights are unknown, complete compromise is 
possible (in linear time) provided one value in the data base 
is known. But compromise is impossible without initial in¬ 
formation—even if overlap between queries is unrestricted. 

Chin showed how achieving compromise with linear sys¬ 
tems applies to characteristic-specified queries. 9 His model 
assumes only that no two individuals belong to the same 
categories. 


Selection methods 

Dobkin, Jones, and Lipton, 6 Davida et al., 10 and Reiss 11 
have considered key-specified queries which select some 
element from the query set; e.g., the median, largest, or 
smallest data value. Their results show that using combi¬ 
natorics, compromise is generally achievable in linear time 
(or better) provided that no two individuals have the same 
data value. For example, consider these two queries involv¬ 


ing records with distinct overdraft values: 

MEDIAN(A,D,E; OVERDRAFTS)=2 

MEDIAN(B ,D ,G; OVERDRAFTS)=2 

Since D is the only individual common to both queries, the 
returned median value 2 must be the number of D’s over¬ 
drafts. DeMillo, Dobkin, and Lipton also showed that this 
method of compromise applies even if the database “lies 
about the median value, provided it returns with some value 
in the query set.” 12 For example, the previous example 
would result in a compromise of D’s overdrafts even if 2 
were not the true median of the query set values. 


PRIVACY SAFEGUARDS 

We have considered two measures—restricting the size of 
allowable query sets and restricting the amount of overlap 
between queries—which appear on the surface to reduce the 
threat of compromise. Yet on closer inspection, we observed 
that both of these measures fail to defend a system from 
even simple attacks. We shall now examine other privacy 
safeguards which will, hopefully, prove more effective. 

Completely secure data bases 

Several studies have reported conditions which, if im¬ 
posed on the structure or contents of a database, guarantee 
its security. 6,7 ’ 9 ’ 13,14 ’ 15 In most cases, these conditions depend 
on the users having little or no supplementary knowledge 
about either the structure or the contents of the records. 
This generally rules out the users themselves (or their 
friends) being represented in the database. Since these con¬ 
ditions are seldom met, they seem to have little value in 
practice. 

Random subfiles 

Rather than making the entire Census available to statis¬ 
ticians, the Census Bureau distributes random subfiles of 
the data. The 1960 Census, for example, was distributed on 
tape as a Viooo sample of the full Census with names and 
addresses removed. Also deleted was geographic informa¬ 
tion below the level of broad city-size class within the nine 
geographical divisions of the country. 16 As a result, the pay¬ 
off from an attempted compromise of a particular individ¬ 
ual’s privacy from Census tapes is sufficiently small to not 
pose a serious threat. Unfortunately, this technique is ap¬ 
plicable only to large databases; other methods are needed 
to prevent compromise of small databases. 

Rounding schemes 

Several studies have been made of rounding schemes for 
modifying the answers to queries. 3,16-21 One such approach 
is pseudo-random rounding. Truly random rounding is not 
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secure since the correct answer to any query can always be 
determined by averaging a sufficient number of responses 
to the same query. With pseudo-random rounding, the same 
query always returns the same response. This method ap¬ 
pears to be reasonably effective, although any kind of “sto¬ 
chastic error” added to responses is subject to removal by 
methods from communication theory. 

A second approach is to always round the actual response 
down to the nearest multiple of some integer or to report a 
range. For example, the database could respond to the query 
COUNT(LAWYER; OVERDRAFTS) with 50 (or the range 
50-60) rather than the true value 56. However, as noted by 
Karpinski, this method is easily circumvented if records can 
be added to the database. 18 To determine the true value of 
the above query, an intruder could simply add records to 
the data base for fictitious lawyers having one overdraft 
each; after the 4th record is added, the response becomes 
60 (or the range 60-70), implying an original value of 56. 
Even if records cannot be added to the database, it may be 
possible to modify the characteristic of the query to include 
records of individuals for whom the number of overdrafts is 
known. 


Partitioned databases 

Another approach to preventing compromise partitions 
the database into groups. In the Yu and Chin scheme, 22 
queries must be for characteristics involving entire groups, 
making it impossible to isolate a particular individual. For 
example, the sample database could be partitioned by 
profession into four groups: 


Group Identifying Characteristic Members 


1 LAWYER A,E,F,H 

2 DOCTOR D,I 

3 JOURNALIST+ 

BUDGET DIRECTOR B,K,L 

4 PRESIDENT+SENATOR C,G,J 


Whereas queries involving complete groups (e.g., all lawyers 
and doctors) are allowed; queries involving only part of a 
group (e.g., all male lawyers) are not. Clever query se¬ 
quences can at best disclose information about an entire 
group. Yu and Chin show that the technique can be made 
effective even if the database is dynamically undergoing 
insertions, deletions, and updates. 

The drawback is that the partitions may destroy the use¬ 
fulness of the database. If the groups are not properly 
formed, users may not be able to obtain needed information. 
Whether or not a useful database can be partitioned in this 
way is open. 

Threat monitoring 

Threat monitoring techniques detect the possible occur¬ 
rence of compromise. Felligi showed that it is at least the¬ 
oretically possible to determine whether the response to a 
query, when correlated with the responses to earlier queries, 


could result in compromise. 23 However, the method is too 
cumbersome to apply in practice. Hoffman and Miller sug¬ 
gested that a log or audit trail of queries be kept and in¬ 
spected for unusual bursts of activity or queries returning 
small counts. 1 Although certain compromises may go un¬ 
detected, the presence of a log may provide a deterrent. 

Schlorer suggested that frequency counts of categories be 
used to determine whether or not a given query might lead 
to compromise because of small counts. 21 Response is not 
made to any query involving categories c x , . . . , c k unless 
the product of the frequency counts COUNT(Ci)/n (for i=l, 
. . . , k) is above some threshold. _ 

CONCLUSIONS 

The “obvious” techniques for reducing the threat of com¬ 
promise—e.g., limiting the range of allowable responses, 
restricting the amount of overlap between query sets, and 
certain rounding schemes—are easily circumvented. Other 
techniques—e.g., partitioning—may be robust, but at the 
price of limiting the usefulness of the data base. The con¬ 
clusion is that complete privacy cannot be enforced without 
severely restricting the free flow of information. The ques¬ 
tions of interest then become: 

• Can we measure the relative security of a data base? 

• What is an acceptable level of security? 
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INTRODUCTION 

In the last several years, methods for controlling security in 
computer systems have become widely known. While in 
1972 there were only one or two books and three bibliog¬ 
raphies on computer security and privacy, in 1977 there were 
at least 15 books and six bibliographies on the topic. Ex¬ 
panding public concern with the problem is evidenced by a 
great deal of federal, state and local legislation. 1 ’ 2 Govern¬ 
mental agencies such as the National Bureau of Standards, 
the Defense Department, and the National Science Foun¬ 
dation are all sponsoring research efforts in the area, as have 
some private manufacturers. 3-5 

One consequence of this work is that a significant part of 
the computing community is now aware of techniques 6 for 
maintaining security in computer systems. Unfortunately, 
the question of how to measure the costs and effectiveness 
of the various security methods is still largely unexplored. 
Only recently has there been any reliable work done on 
costs of privacy transformations or authentication methods, 
and researchers have only scratched the surface in investi¬ 
gating metrics for security systems. 7 While some more for¬ 
mal work has begun, 8-13 there has so far been very little 
useful application of this theoretical work to practical se¬ 
curity decisions. One exception is a privacy cost model 
which has been recently introduced. 14 This work assesses 
the impact of various privacy regulations upon data proc¬ 
essing installations which handle personal information. 
Goldstein discusses conversion costs to meet privacy re¬ 
quirements and the “privacy increment” to installation op- 

* Work supported in part by Grant MCS76-09214 from the National Science 
Foundation and by the University Committee on Research, The George 
Washington University. 


erational costs. He does not, however, deal directly with 
the effectiveness of security techniques. 

We will address the question of security effectiveness. 
We here describe SECURATE, a computer installation se¬ 
curity evaluation and analysis system, which is based upon 
a model of a computer installation as a set of triples com¬ 
posed of objects, threats, and security features. SECUR¬ 
ATE uses security rating functions based on fuzzy set theory 
to provide security ratings for an installation as a whole as 
well as for subsections. It helps to determine weak and 
strong points and facilitates the comparison of alternative 
security designs. SECURATE is not meant to be a substitute 
for a human decision maker. 

The SECURATE user first supplier a set of object-threat- 
feature triples necessary to describe a computer system from 
a security point of view. OBJECTS are defined as the re¬ 
sources within a computing system, the loss of which would 
have a cost to the owner. THREATS are activities which a 
potential intruder may employ to gain unauthorized access 
to an object. This term also refers to chance events which 
may jeopardize an object. FEATURES are protective meas¬ 
ures which present some degree of resistance to a threat. 
The second section describes this framework in more detail. 

After the system description is complete, the user is asked 
to specify his or her outlook on security; some view total 
system security as an all-or-nothing thing, while others see 
it as dependent in various ways on a system’s individual 
components. Based upon the user’s outlook and the system 
description, SECURATE supplies evaluation functions 
which return natural-language security ratings of the system 
or its subsystems. The fifth section describes these evalua¬ 
tion functions. 

After the system was implemented in APL, it was used at 
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seven installations by students who were doing risk analyses 
of the installations. Feedback from this initial group of users 
is discussed in the seventh section. 

TECHNICAL BASIS 

The technical basis for SECURATE is in earlier work 15 
which defines a security system as a set of objects, each 
with a loss value; a set of threats, each with a likelihood; 
and a set of features, each with a resistance. This work 
addresses the problem of imprecision in the approximation 
of values, likelihoods, and resistances by using linguistic 
variables 16 to combine the values of these variables into 
security ratings. 

Clements’ model groups resources within computing sys¬ 
tems which are vulnerable to some security threat as the set 
O of security objects. Each object in the set possesses a loss 
value to its owner. 

Associated with each security object Oj is a number of 
potential intrusion activities; all of these together form the 
set T of security threats. Each threat T { has associated with 
it a likelihood of occurrence. 

The object-threat relations form a bipartite directed graph 
(Figure 1) in which edge TjOj exists only if threat T t is a 
viable means of compromising object O s . The relations of 
threats to objects is not one to one; a threat may compromise 
any number of objects and an object may be vulnerable to 
more than one threat. 

Finally, there is the set F of security features. A security 
feature performs a “firewall function” by presenting some 
degree of resistance to a penetration attempt. The set of 
security features transforms the bipartite graph of Figure 1 
into the tripartite graph of Figure 2. In a “protected” system 
all edges are of the form T t F k and F k O } . Any edge of the 
form T t Oj identifies an unprotected object. Each triplet (T f , 
F k , Oj) in TxFx O contributes to the overall security rating 
of a system, as we shall see below. 

In attempting to specify the object values, threat likeli¬ 
hoods, and feature resistances, one is confronted with the 
problem of imprecision. In evaluating a computer system’s 
security, one generally relies on imprecise human judgment 
to provide approximate measures of the effectiveness of 
security features. The problem is aggravated when one at¬ 




tempts to produce security ratings for entire systems from 
these measures. The assignment of a numerical security 
rating is inconsistent with the complexity of data processing 
installations and the imprecision of the underlying measures. 
Stating that an installation is “.65 secure” has limited appeal 
for imparting a sense of how secure an installation is. In 
addition, the precision implied by such a rating should be 
viewed skeptically. 

Clements suggested that it is possible to make meaningful 
measurements of the security of a computer system through 
the use of linguistic variables—variables which assume val¬ 
ues which are words rather than numbers. Using this ap¬ 
proach, the object values, threat likelihoods, and feature 
resistances (as well as the resultant security rating) are spec¬ 
ified in terms such as high, low, and medium. Appropriate 
modifiers provide finer resolution by allowing terms such as 
very high, somewhat low, etc. 

The user assigns linguistic values {high, medium, very 
high, etc.) to the component variables (loss value, likeli¬ 
hood, resistance) for each (object, threat, feature) triplet in 
the system. These measures determine the contribution of 
that particular triplet to total system security. How this is 
done is discussed in a later section. 

Each linguistic variable is a fuzzy set whose members are 
real numbers in the interval [0,1]. These values comprise 
the compatibility function, p f , for the specific linguistic var¬ 
iable. For example, if /x h i g h(0.8)=0.9, 0.9 represents the de¬ 
gree to which a non-fuzzy rating of 0.8 on a base scale 
agrees with a fuzzy rating of high. Figure 3 illustrates what 
the complete compatibility functions for typical high and 
very high terms might be. More detail on linguistic variables, 
base scales, and compatibility functions can be found in 
Reference 16. 


OBJECT HIERARCHY AND LINGUISTIC TERMS 

SECURATE provides a “default” hierarchical structure 
of objects commonly found in computer installations. 17 A 
small part of this object hierarchy and the corresponding 
threats and features are presented in the appendix. 

There are two phases involved in using the system: (1) 
typing in a description of the installation and (2) using the 
security analysis functions. 

Object value, threat likelihood, and feature resistance for 
each security point of interest are first specified by the user 
in terms of linguistic variables. The vocabulary and Backus- 
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Figure 3a—Typical compatibility function of “high” probability 



Figure 3b—Typical compatibility function of “very high” probability 


Naur form syntax of the rating language, along with exam¬ 
ples, are shown in Figure 4. 

Once the installation to be analyzed is described in terms 
of these triples, the functions described in the fifth section 
are invoked by the user to evaluate and analyze its security. 

THE USER INTERFACE 

From the start of the SECURATE project, an important 
objective was to design and implement the system so that 
it would be as hospitable to the users as possible. Our goals 
concerning user-oriented features were primarily to keep the 
system simple, easy to use, and non-tedious. More specifi¬ 
cally, we were concerned with the following points: 

(a) User Understanding—for obvious reasons, achieving 
adequate user understanding is very important. Not 
only won’t the system be useful if the user doesn’t 
understand it, but it won’t be used. 

(b) Simple, Non-tedious Interface—a similar, much sim¬ 
pler system had been developed earlier by a student 
as a term project. A unanimous criticism of that sys¬ 
tem was that it took too long to use and the data entry 
was too tedious. As our system was to require con¬ 
siderably more information, it seemed important to 
keep the interaction as short, concise, and painless as 
possible. 


(c) Useful Evaluation Functions—while it may seem that 
this is the most important point, it may actually be the 
least. A system which a user understands and is com¬ 
fortable using is more likely to be used and be helpful 
than a system that doesn’t possess these qualities, 
even if the evaluation functions provided by the first 
aren’t quite as useful as those provided by the second. 

Since the object hierarchy was used as a guide for col¬ 
lecting user data, it provided a convenient basis for struc¬ 
turing the input. We drew up forms which corresponded in 
format exactly to what appeared on the screen. The user 
wrote down on the forms only the necessary information 
and then transferred it easily to the system. These forms, by 
integrating the system commands with the input data in a 
coherent way, familiarized the users with the system’s op¬ 
eration prior to their using it. Figures 5a and 5b show an 
example of the input form and the corresponding data entry. 
Further information concerning the user interface is given 
in Reference 18. 

SECURATE assumes that the installation will be similar 
to that modeled by the hierarchy in Appendix A, although 
the user can modify the hierarchy appropriately as he or she 
supplies the triples information. SECURATE leads the user 
through the hierarchy (and any user modifications to it), 
providing the opportunity at each node to add offspring or 
specify triples. If a triple is specified for an object with 
offspring, it is assumed to refer to that object and each of 
its offspring. Objects, threats, and features appearing in 
more than one triple may have different values, likelihoods 
or resistances, respectively. 


(sentence) "= (compound phrase) | (simple phrase) 
(compound phrase) :: = (conjunctive phrase) | (range phrase) 
(simple phrase) "= (relational phrase) | (hedged primary) 
(conjunctive phrase) ” = (relational phrase) AND (relational phrase) 
(range phrase) "= (hedged primary) TO (hedged primary) 
(relational phrase) :: = (composite relation) THAN (hedged primary) 
(composite relation) ::= (relation hedge) (relation) | (relation) 
(relation hedge) := NOT | MUCH | SLIGHTLY 
(relation) :: = LOWER | HIGHER 

(hedged primary) ." = (hedge) (primary) | (primary) | (fuzzy number) 
(hedge) ::= NOT | VERY | MOREORLESS | QUITE | PRETTY | 
SORTOF | REALLY | EXTREMELY | INDEED 
(primary) :: = LOW | HIGH | MEDIUM 
(fuzzy number) ::= (fuzzifier) (number) 

(fuzzifier) ::= ABOUT 
(number) ::-l|2|3|4|5|6|7|8|9|10 
Figure 4a—BNF grammar of rating language 

high 

low 

medium 
not high 

moreorless medium 

indeed low 

low to medium 

(about 4) to about 6 

slightly lower than pretty high 

not higher than medium 

(much higher than low) and slightly lower than sortof high 
Figure 4b—Examples of terms available in rating language 
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OBJECT NO: 

ADD, A name or number 
VALUE, V object value 
THREAT NO THREAT LIKELIHOOD 


1 _ 

A METERING EQUIPMENT 


FEATURE NOS FEATURE RESISTANCE 


OBJECT NO: 

ADD, A name or number 
VALUE, V object value 
THREAT NO THREAT LIKELIHOOD 
8 MEDIUM 

10 PRETTY LOW 


V VERY HIGH _ 

FEATURE NOS FEATURE RESISTANCE 
2 PRETTY HIGH 

.29 30 MEDIUM 


Figure 5a—Data input form 


ENTER THE OBJECT NUMBER FOR THE NEXT OBJECT : 

1 

HARDWARE 

ADD METERING EQUIPMENT 

METERING EQUIPMENT RECEIVED OBJECT NUMBER 71 
O 

OBJECT NO 11, CENTRAL MACHINE IS NEXT . 

V VERY HIGH 

THREAT NO THREAT LIKELIHOOD FEATURE NOS FEATURE RESISTANCE 

8 ~MEDIUM 2 PRETTY HIGH 
—> 

10 PRETTY LOW 29 30 MEDIUM 


Figure 5b—Data entry information typed by the system is underlined 


Once the triples are entered, they may be printed out 
using the DISPLAY function. For each triple this prints out: 
the triple number; the object name, number, and value; the 
threat name, number, and value; and the feature resistance. 
Figure 6 shows the DISPLAY output for a sample system. 
The information describing the installation is automatically 
saved and may be used later with repeated applications of 
the system. 

EVALUATION FUNCTIONS 

The general system structure is shown in Figure 7. There 
are several security evaluation functions implemented. Each 
produces an evaluation based upon the section of the system 
being considered and upon a given outlook on security. 

The possible sections of a system which can be evaluated 
are the following: 

(a) Overall System—This function returns a security rat¬ 
ing for the entire installation. That is, it rates the entire 
set of triples. 

(b) All subsections of a Section—with either the top level 
of the installation hierarchy or one of the subsections 


specified, this function returns an individual rating for 
each subsection at the next lower level. For example, 
if in our sample system, the top level of the hierarchy 
was specified for a sectional analysis, security ratings 
would be printed out for each of the following sub¬ 
sections: hardware, software, and the computer center 
(Figure 8). 

(c) Individual Subsection—a security rating is returned 
for a specified subsection of the installation. Only 
triples for that subsection (including offspring) are 
considered (see Figure 9). 

In choosing an outlook, the user in effect describes how 
he or she views security. Once an outlook is chosen, it stays 
in effect for all of the evaluation functions until it is respec¬ 
ified. 

The outlooks are the following: 

(a) Weakest Link—this will look for the weakest feature 
resistance and return that as the security rating. The 
theory here is that the system is only as secure as its 
weakest link (Figure 10). 

(b) Selected Weakest Link—this produces a weakest link 
rating based on those triples which satisfy the condi- 
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DISPLAY 


FOLLOWING IS A LIST OF OBJECTS ADDED, THEIR ASSIGNED OBJECT 
NUMBERS, AND THEIR PARENT IN THE HIERARCHY: 

OBJECT OBJECT NO PARENT 


METERING EQUIPMENT 


71 


1 


OBJECTS 


THREATS 


FEATURES 


TRIPLE 

NO 


k NUMBER 

k 


NAME 

VALUE 


NUMBER 


NAME 

LIKELIHOOD 


NUMBER 


NAME 

RESISTANCE 


kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk 


kkk 

★ 11 CENTRAL MACHINE 
k VERY HIGH 

•kick 

k 11 CENTRAL MACHINE 

k 

k VERY HIGH 

kkk 

k 12 STORAGE MEDIA 

k 

k HIGH 

kkk 

k 12 STORAGE MEDIA 
k HIGH 

kkk 

k 71 METERING EQUIPMENT 
k LOW 

kkk 

k 22 PROGRAMS 
k MEDIUM 

kkk 

k 23 DATA 

k 

k HIGH 

kkk 

k 23 DATA 

k 

k HIGH 

kkk 

k 23 DATA 

k 

k 

k HIGH 


kkk 

k 8 UNAUTHORIZED USE 
k MEDIUM 

kkk 

k 10 HUMAN ERROR 

k 

k PRETTY LOW 

kkk 

k 13 UNAUTHORIZED READ 

k 

k HIGH 

kkk 

k 11 THEFT 
k LOW 

kkk 

k 4 HARDWARE TAMPERING—MODIFIED 
k LOW 

kkk 

k 46 INADEQUATE DEBUGGING 
k FAIRLY HIGH 

kkk 

k 20 UNSECURED STORAGE MEDIA 

k 

k HIGH 

kkk 

k 33 EXPOSED OUTPUT 

k 

k MEDIUM TO HIGH 

kkk 

k 43 DATA PREPARATION ERRORS 

k 

k 

k PRETTY HIGH 


kkk 

k 2 GUARD 
k PRETTY HIGH 

kkk 

k 29 OPERATOR TRAINING 
k 30 DETAILLED, ACCURATE, ACCESSIBL 
k MEDIUM 

★★★ 

★ 43 DATA ENCRYPTION 
k 44 EFFECTIVE STORAGE ACCESS CONTR 
k PRETTY LOW 

kkk 

k 31 PHYSICAL ACCESS CONTROLS 
k FAIRLY HIGH 

kkk 

k 21 LOCKS AND ALARMS ON MACHINE CO 
k HIGH 

kkk 

k 114 PROGRAM TESTING AND VALIDATION 
k (FAIRLY LOW) TO MEDIUM 

kkk 

k 60 ADEQUATE AND ENFORCED LIBRARY 
k 61 USAGE LOG 
k PRETTY LOW 

kkk 

k 90 CLEAN DESK POLICY 
k 91 USER EDUCATION 
k LOW 

kkk 

k 103 SECOND PERSON VERIFICATION 
k 104 CHECKSUMS 
k 105 SOFTWARE CHECKS 
k HIGH 


Figure 6—Display of Triples of the Sample System 



_Data flow 

- Function calls 


tion that either the object value or the threat likelihood 
is greater than a user specified minimum. The theory 
here is that one would only want to consider triples 
where the object is of at least a certain value or the 
threat is of at least a certain likelihood (see Figure 8). 

(c) Fuzzy Mean—this performs a fuzzy mean 15 on the 
feature resistances and returns the result as the rating. 
The theory here is that a system’s security is the mean 
of the security of its components (Figure 11). 

(d) Weighted Fuzzy Mean—this performs a fuzzy mean 
on the feature resistance weighted by the greater of 
the object value and threat likelihood for each triple. 
The theory is that of (c), with the additional assump¬ 
tion that the more valuable objects and those with 
more likely threats should receive greater weight in 
the security rating (see Figure 12). 

(e) Fuzzy Mean with Each Major Subsection Weighted 


Figure 7—System structure 





536 


National Computer Conference, 1978 


ENTER THE NUMBER OF THE RATING FUNCTION YOU WISH TO USE: 2 
SECTIONALRATING 

ENTER THE PARENT OBJECT NUMBER (0 FOR THE TOP LEVEL IN THE HIERARCHY ): 0 
SPECIFY MINIMUM FOR HARDWARE : MEDIUM 
4 ELEMENTS) USED 

SPECIFY MINIMUM FOR SOFTWARE : HIGH 
1 ELEMENT (5) USED 

SPECIFY MINIMUM FOR THE COMPUTER CENTER : PRETTY HIGH 
4 ELEMENTiS) USED 


★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★ 

★ 

★ NAME RATING (USING SELECTED WEAKEST LINK) 

★ 

★ HARDWARE PRETTY LOW 

★ SOFTWARE PRETTY HIGH 

★ THE COMPUTER CENTER PRETTY HIGH 

★ 

★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★ 


Figure 8—Sectional Ratings for Sample System 


SETRATE 3 
INDIVID UALRA TING 

ENTER THE NUMBER OF THE OBJECT TO BE RATED: 2 


★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★ 

★ 

★ NAME RATING (USING FUZZY MEAN) 

★ 

★ SOFTWARE SORTOF MEDIUM 

★ 

★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★ 

Figure 9—Individual Subsection Rating (Software Subsection) of Sample System 


ENTER THE NUMBER OF THE RATING FUNCTION YOU WISH TO USE: 1 
OVERALLRATING 


★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★•A- 

★ 

★ NAME RATING (USING WEAKEST LINK) 

★ 

★ THE INSTALLATION LOW 

★ 

★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★ 

Figure 10—Rating Using Weakest Link Outlook and Overall Section of Sample System 


SECTIONALRATING 

ENTER THE PARENT OBJECT NUMBER (0 FOR THE TOP LEVEL IN THE HIERARCHY): 0 


★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★ 

★ 

★ NAME RATING (USING FUZZY MEAN) 

★ 

★ HARDWARE ((SLIGHTLY LOWER) THAN FAIRLY HIGH) AND {SLIGHTLY HIGHER) THAN SORTOF HIGH 

★ SOFTWARE SORTOF MEDIUM 

★ 

★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★ 

Figure 11—Partial Rating of Overall Section of Sample System Using Fuzzy Mean Outlook 
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★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★ 

★ 

★ NAME RATING (USING FUZZY MEAN WEIGHTED BY VALUE) 

★ 

★ OPERATING SYSTEM (MOREORLESS MEDIUM) TO (SORTOF HIGH) 

★ PROGRAMS MOREORLESS MEDIUM 

★ DATA SORTOF MEDIUM 

★ 

★ THE LOWEST RATING WAS GIVEN TO: 

★ DATA 

★ 

★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★ 


Figure 12—Sample System Software Sectional Rating Using Fuzzy Mean Weighted by Value Outlook 


by Maximum Object Value—for each major subsec¬ 
tion of the object specified, this finds the fuzzy mean 
of the resistances. It then weights these fuzzy means 
by the maximum object value found in the triples for 
each major subsection and averages these weighted 
means. The theory is similar to (d), but with the as¬ 
sumption that the major subsections should be 
weighted by their relative values, irrespective of the 
number of triples they each have. 

THE USE OF APL 

APL is extremely well suited to applications involving 
linguistic variables and fuzzy set operations. Using appro¬ 
priately named functions and variables, the linguistic vari¬ 
ables can be easily converted into the corresponding base 
variables using the APL “execute” function. For example, 
HIGH might be a vector of (0 0 0 0 0 .1 .5 .9 1), represent¬ 
ing the linguistic variable high. VERY might be a function 
which sharpens the curve given to it as its argument, perhaps 
squaring the argument. Then as shown in Figure 13, if 
VALUE were a variable containing the character string 
“VERY HIGH”, executing it would return the vector 
(0 0 0 0 0 .01 .25 .81 1), representing the base variable for 
the linguistic variable very high (Figure 3 gives the curves 
representing high and very high). The important point here 
is that APL eliminates the need to do any parsing of the 
input values; the linguistic variables input are stored in char¬ 
acter string format and used as input to the scoring functions 
at the appropriate time. Additionally, the built-in APL ma- 


VV£RHnv 

v o UT+-VER Y IN 
[1] OUT+-LN xIN 

V 

HIGH 

0 0 00 0 0.1 0.5 0.9 1 

VALUE 
VERY HIGH 

1-VALUE 

0 0 00 0 0.01 0.25 0.81 1 

Figure 13—APL Execution of Linguistic Variables 


trix operations are well suited to the fuzzy set operators, 
which use vectors and matrices extensively. These operators 
are described in detail in Reference 15. 

On the negative side, APL is interpretive; this makes it 
significantly slower than compiled programs for repeated 
runs. In addition, it is poorly suited to applications not 
involving vectors or arrays. The latter point is important for 
the security evaluation system since most of the code deals 
with the user interface and the analysis functions. Not only 
were these awkward to program, but they run rather slowly 
(these two points not being unrelated). The rating functions, 
however, which make heavy use of the matrix capabilities 
while performing fuzzy set operations, are well suited to 
APL. 


USER REACTIONS 

Shortly after development started, we arranged to have 
the system (running on an IBM 370/145 system under the 
CP/CMS operating system) tested by students who were 
doing risk analyses of computer installations as term proj¬ 
ects. SECURATE was used to analyze seven installations, 
including one at a large bank and one at a large utility 
company. 

From written evaluations solicited from each of the users, 
as well as from conversations with them, it seems clear that 
the system achieved its goal of increasing understanding of 
installation security. In fact, a couple of users remarked that 
just filling out the forms made the strengths and weaknesses 
of an installation’s security a lot clearer. Apparently focus¬ 
ing their thoughts into a logical, well-defined framework 
enabled them to view the situation more clearly and—even 
before using the system—to gain some of the insights we 
had hoped the system would provide. 

The most interesting observations were those concerning 
the use of fuzzy variables. There appears to be definite 
tradeoff between user acceptance and ease of use. The con¬ 
cept of fuzzy variables was new to all of the users and was 
greeted with a certain amount of skepticism. While their 
acceptance of the idea grew as they continued to be exposed 
to it and had experience in using it, some of them remained 
skeptical. On the other hand, some of them commented, and 
we strongly feel this to be true, that the use of these words 
instead of numbers was a definite help in minimizing the 




538 


National Computer Conference, 1978 


tedium involved in collecting the input data. The largest 
installation turned out to be represented by 136 triples, 
which cam to over 300 different measurements the user had 
to make. Pinpointing each one on a scale of 1 to 10 appears 
to us to be a lot more taxing than rating each one as a 
linguistic variable. Although we didn’t do any comparative 
studies (which in retrospect would have been a good idea), 
many users seemed to agree with this in informal discus¬ 
sions. 


SUMMARY 

We have described an interactive security evaluation system 
which uses fuzzy metrics. The system models the installa¬ 
tion to be analyzed as a set of object-threat-feature triples. 
The associated measures—object values, threat likelihoods, 
and feature resistances—are then used as input to security 
evaluation functions. The user specifies these features in 
terms of “fuzzy” linguistic variables. The system, imple¬ 
mented in APL, is currently operational on an Amdahl 470 
computer. 
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APPENDIX—THE OBJECT HIERARCHY, THREATS, 
AND FEATURES 

The structure of the threats is based on the object hier¬ 
archy, which is used as an outline. Threats are listed after 
the objects they refer to, the objects being specified by name 
and number from the object hierarchy. A threat listed after 
a non-terminal node of the object hierarchy refers to all 
objects descending from the node. Features which counter 
threats follow the listing of the threats. 

Partial object hierarchy 

2. Software 

2.1 Operating system 

2.2 Programs 

2.2.1 Applications 

2.2.1.1 Source 

2.2.1.2 Non-source 

2.2.2 Contract programs and packages 

2.2.3 System utilities 

2.2.4 Test programs 

2.3 Data 

2.3.1 Personal data 

2.3.1.1 Payroll 

2.3.1.2 Personnel 

2.3.1.3 Other personal data (Privacy Act of 
1974, §3(a)(4)) 

2.3.2 Institution data 

2.3.2.1 Marketing 

2.3.2.2 Financial 

2.3.2.3 Operations 

2.3.2.4 Planning 

2.3.2.5 Other 


Partial Threats Listing 
2. Software 

16 A. Unauthorized access: R/W/E 

17 Modification of operating system and system rou¬ 
tines 

18 Inadequate controls on I/O facilities 

19 Password compromise 

20 Unsecured storage medium 

21 Access outside of allocated memory 

22 Modification of stored state vector 
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23 Unauthorized CE activity 

24 Line tapping and spoofing 

25 Erroneous or inadequate usage of protection fa¬ 

cilities 

26 B. Unauthorized access: read 

27 Extra copies of output printed 

28 duplicates printed 

29 printing restarted before end 

30 Use of erroneous distribution labels 

31 Use of erroneous distribution lists 

32 Theft of mail 

33 Exposed output 

34 in user possession 

35 within distribution system 

36 at operator’s console 

37 work in progress 

38 Unauthorized reading of terminal buffers 

39 Indirect exposure of output 


40 C. Unauthorized access: write 

41 Modification or spoof of mail transactions 

42 Unauthorized modification of data during prepa¬ 

ration 

43 Data preparation errors 

44 Modification of original written data input 

2.1 Operating system 

45 Defective implementation 

2.2 Programs 

46 Inadequate debugging 

47 Incomplete operation specifications 

48 Inadequate or erroneous error handling 

49 Exposure following abnormal end 

50 Improper operation 

2.2.2 Contract programs and packages 

51 Dishonest programs 
2.2.4 Test programs 

52 Unexpected alteration of real data 


Features 


Feature No. 

Threat Numbers 

Feature Name 

46 

16 

Effective authorization and access control mechanism 

47 


Minimum authorization policy 

48 

17 

Effective authorization and access control mechanism 

49 


Minimum authorization policy 

50 


Dual authorization required for changes 

51 


Super user authorization required for changes 

52 


Log of attempted violations 

53 

18 

Self-modifying I/O routines not allowed 

54 

19 

Direction in password choice 

55 


Store in encrypted form 

56 


Automatic delay after invalid login attempt 

57 


Encrypted transmissions to terminals 

58 


Use of interactive authentication procedure 

59 

20 

Adequate access controls 

60 


Adequate and enforced library facility 

61 


Usage log 

62 


Proper labelling 

63 

21 

Proper system design 

64 


Effective authorization and access control mechanism 

65 


Adequate I/O controls 

66 


Protection of state vector 

67 

22 

Storage in protected storage 

68 

23 

Administrative controls 

69 


Human verification 

70 


Supervision 

71 


Limited CE access 

72 

24 

Encryption 

73 

25 

Effective human engineering 

74 


Clear, easy to use protection facilities 

75 


Adequate documentation 

76 


User education 

77 

26 

(See features for threats 27-39) 

78 

27 

Print log 

79 


Security conscious I/O routines 
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80 

28 

Print log 

81 

29 

Print log 

82 


Security conscious I/O routines 

83 

30 31 

Careful administrative procedures 

84 

32 

Careful administrative procedures 

85 


Important mail sent registered or by courier 

86 


Delivery confirmation 

87 

33 

Trace log of sensitive output 

88 


Library facility for sensitive output 

89 


{See also features for threats 34-37) 

90 

34 

Clean desk policy 

91 


User education 

92 

35 

Guarding work in transit 

93 

36 

(Refer to features for threats 1-18) 

94 

37 

Guarding work in progress 

95 

38 

Buffer erase mechanism 

96 

39 

Paper shredder 

97 


Use of old ribbons for sensitive jobs 

98 


Destruction of carbon paper and ribbons 

99 

40 

(Refer to features for threats 41-44) 

100 

41 

Careful administrative procedures 

101 


Important mail sent registered or by courier 

102 


Delivery confirmation 

103 

42 43 

Second person verification 

104 


Checksums 

105 


Software checks 

106 

44 

Verification checks 

107 


Checksums 

108 


Software checks 

109 


Originator verification 

110 

45 

Testing 

111 


Audit programs 

112 


Testing and verification 

113 


Penetration attempts 

114 

46 

Program testing and validation 

115 

47 

Adequate documentation and design specs 

116 

48 49 50 

Adequate documentation and design specs 

117 


Program testing and validation 

118 


Programmer education 

119 

51 

Program testing and validation 

120 


Code inspection, recompilation 

121 


Choosing writer who could not benefit 

122 

52 

Testing on setup data 

123 


Containment of test programs 




A panel session—The contribution of planning to information 
systems productivity 


SESSION CHAIRMAN—CHARLES C. TUCKER 

Twentieth Century-Fox Film Corporation 


Panel Members 

John A. Zachman—International Business Machines 
Daniel S. Appleton—Borg-Warner Corporation 
Michael J. Kirrene—Avco Financial Services 


PANEL OVERVIEW—Charles C. Tucker 

The traditional approach to information systems devel¬ 
opment has been to plan at the project level. This leads to 
the development of independent systems, each with its own 
data, for each user. Most organizations suffer from the fol¬ 
lowing shortfalls that results from this traditional approach: 

1. Systems which are not supportive of business plans 

2. Systems that are not easily adaptable to organizational 
changes 

3. Data which is redundant and inconsistent 

4. Systems that are difficult to interconnect 

5. “Squeaky wheel” priority determination. 

Management views this in terms of systems with short 
useful lives, which are expensive to operate and maintain, 
and support areas with little impact on the overall success 
of the company. 

The Business Systems Planning methodology used at Fox 
overcomes these shortfalls by: 

1. Tying information systems plans directly to business 
plans 

2. Designing systems around business processes so they 
are organizationally independent 

3. Treating data as a company resource so it can be shared 
and managed 

4. Laying out the overall architecture or network first so 
the individual systems can be easily interfaced as they 
are developed 

5. Providing a logical framework for determining imple¬ 
mentation priorities. 

Because of our investment in Business Systems Planning, 
we expect that the information systems we develop in the 
future will last longer, cost less to develop and operate, 


support business plans better, and address our most impor¬ 
tant information requirements first. 


THE INFORMATION SYSTEMS MANAGEMENT 

SYSTEM: A FRAMEWORK FOR INFORMATION 

SYSTEMS PLANNING—John A. Zachman 

Technology is precipitating a change in management’s 
perception of the role of information systems (I/S) in the 
business environment. Business Systems Planning is an an¬ 
alytical methodology that has evolved in an attempt to better 
define this role and manage I/S more effectively in support 
of management. 

As a result of a number of Business Systems Planning 
analyses, it is becoming apparent that the key to increasing 
I/S productivity in light of the new role being defined, is the 
recognition of several components of the overall I/S Man¬ 
agement System that must be managed uniquely. 

These components include: 

1. A Strategic Plan and planning process that looks ex¬ 
ternally at the business, as well as integrating with the 
more typical Management Control and Operational 
Control plans and planning processes which look in¬ 
ternally at I/S. 

2. An Information Architecture that depicts the business 
and serves as the “preliminary design” of the inte¬ 
grated product that the I/S organization is in place to 
build. 

3. A Data Architecture (and its attendant management fa¬ 
cility) that protects the consistency of data for man¬ 
agement’s measurement purposes, as well as controls 
the project development activities to insure their con¬ 
formance to the Information Architecture. 

These components of the I/S Management System are 
over and above the traditional Development and Operations 
components, well recognized as requiring unique attention, 
supporting the role of I/S as it is perceived in today’s typical 
environment. The recognition and management of the ad¬ 
ditional components will have a substantial impact in in¬ 
creasing productivity by minimizing systems redesign for 
data integration purposes, minimizing maintenance activity 
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caused by the interdependence of individual systems and 
files, and insuring that the systems developed more closely 
meet the expectations and requirements of management. 

A DATA MODEL APPROACH TO BUSINESS 

SYSTEMS PLANNING AND CONTROL— 

Daniel S. Appleton 

The management problem is to establish a viable Business 
Systems Plan based on the basic business mission. The MIS 
problem is to implement the Business System Plan. To do 
this, MIS must create a system structure in the business 
which will respond to constant changes. MIS must under¬ 
stand the fundamental structure and keep that structure flex¬ 
ible in response to the business. 

At Byron Jackson Pump (BJP) we first developed a data 
model of our business structure.* Next, we tested this model 
conceptually to determine how it would respond to various 
changes: e.g., decentralization, product standardization, 
new product development, productivity improvement, cost 
reduction. 

The data model of the business was then established as 
our basic Management Systems Technology and placed 
under control of what we call the Storage and Processing 
Control System (SPCS). The SPCS provides centralized, 
standardized control over all data elements, internal proc¬ 
essing routines, minimum edits, security routines, and phys¬ 
ical file structures. It defines the basic management tech¬ 
nology which can be customized to the requirements of the 
various BJP manufacturing locations and DP Service Cen¬ 
ters worldwide. It handles the basic spectrum of manufac¬ 
turing systems, from process control through job shop con¬ 
trol, including any hybrid mix of the two, and accommodates 
fluctuations based on product mix changes and strategic 
plans. 

The level of management technology of the SPCS is con¬ 
trolled by the MIS Steering Committee, which funds major 
modifications. Minor modifications required to customize 
for efficiency or effectiveness at a given location are con¬ 
trolled by the location’s System Management Team. 

BJP’s two other MIS control systems, i.e., the Input and 
Output Control Systems, provide the basic tools for distri¬ 
bution, implementation, and customization of the manage¬ 
ment technology of the SPCS at individual locations. The 
I/O Control Systems are managed by DP Service Centers in 
coordination with local Systems Management Teams in each 
location. 

All three control systems taken together comprise BJP’s 
Functional Network (FN) concept. FN is an approach to 
distributed data processing which concentrates on improving 
the efficiency and cost-effectiveness of the three basic DP 
functions: (1) Storage and Processing, (2) Input, and (3) 
Output. Its advantages over the traditional Applications 
Network concept include: 

1. Lower DP marginal costs 

2. Increased compatibility with business growth and dy¬ 
namics 


* See “An Approach to Manufacturing Automation,” Datamation, Oct. 1977. 


3. Better control over DP technology 

4. More efficient use of DP overhead 

5. More effective use of direct DP expenditures 

6. Easier to customize 

7. Reduced implementation time 

8. Higher quality systems 

9. Easier to maintain systems. 

In summary, the data model describing BJP’s management 
system technology provides a consistent, comprehensible 
foundation for development of the business system struc¬ 
ture. Through BJP’s Functional Network, the development, 
maintenance, implementation, customization and distribu¬ 
tion of that management systems technology is controlled 
based on the specific needs of individual locations. 


INFORMATION SYSTEMS PLANNING IN THE NON¬ 
PLANNING ENVIRONMENT—Michael J. Kirrene 

Avco Financial Services (AFS) is the third largest con¬ 
sumer finance company in the U.S. with sizable operations 
in Canada and Australia and footholds in Puerto Rico, Great 
Britain, and Japan. The management team is young and 
aggressive, operations oriented, and determined to be num¬ 
ber one. By any measures AFS is a successful company. 

From a planning point of view, however, AFS represents 
the “typical” organization as defiend by John Zachman. 
Corporate planning and controlling efforts are focused upon 
individual projects. The formal Business Systems Plan as 
defined in the literature does not exist, and attempts to 
develop one would be viewed as bureaucratic endeavors 
designed to further delay addressing immediate data proc¬ 
essing needs. While a company can obviously exist without 
a formal plan, the Information Systems Department must 
have a plan tied to the business goals in order to maximize 
its contribution to the welfare of the company it serves. It 
is even possible with todays operations support systems that 
data processings activities (or lack of activities) can be det¬ 
rimental to the business. 

The challenge is how to develop effective planning in a 
non-planning environment. Our Information Systems De¬ 
partment has taken a unilateral approach, including senior 
management whenever and wherever possible, but always 
taking the initiative. The results have been mixed. Although 
we do not have a Business Systems Plan, we at least have 
a projects plan reviewed by senior management on a quart¬ 
erly basis and, more importantly, have taken that first cru¬ 
cial step of planning together with the company. 

Intellectually, it is easy to accept the need for a Business 
Systems Plan properly derived from a careful analysis of the 
functional processes and information flow of the business. 
In the real world, however, it appears that rather than at¬ 
tempting to sell the typical company on Business Systems 
Planning, the Information Systems Department should be 
acting as a catalyst, developing business planning sponsors, 
aligning data processing more and more with the business 
objectives by learning more about the business it serves, 
and aggressively participating in the evolution from project 
planning to Business Systems Planning. 



A panel session—Organizational factors in the 
allocation of computing resources 


SESSION CHAIRMAN—ROB KLING 

University of California, Irvine 


Panel Members 

William Dutton—University of California, Irvine 
James Danziger—University of California, Irvine 
Einar Steffereud—Network Management Associates 
J. A. Sutton—IBM Corporation 

A POLITICAL PERSPECTIVE ON COMPUTERS IN 

LOCAL GOVERNMENT—William Dutton 

Most people concerned with computing in complex or¬ 
ganizations do not consider computers a subject for political 
analysis; but few people can maintain this view in the future. 
Social scientists define “politics” as the allocation of goods, 
services, values, and resources. Thus every organization is 
faced with “political” decisions when deciding whether to 
fund one program rather than another. From this point of 
view, “politics” is neither inherently “dirty” nor restricted 
to institutionalized procedures, such as voting, but is a com¬ 
mon element of organizational life. Computing is a political 
technology because it can, and frequently is, used to effect 
the distribution of values among various interests. In any 
organization there are serious differences of opinion and 
interest over the adoption, support, use, and control of com¬ 
puter technology. 

The political elements of computing can be addressed in 
three areas: (1) the allocation of computing resources as a 
means to control other related resources (e.g., staff); (2) the 
computer’s use to effect the distribution of information, and 
therefore the distribution of power and control in an organ¬ 
ization, and (3) the computer’s use to reinforce an organi¬ 
zation’s political economy as reflected in the kinds of goods 
and services it provides. This talk draws upon data collected 
by the URBIS Group in several national surveys and case 
studies of computing in American local governments since 
1974. 

We have found computing to be a complex resource 
“package.” And the allocation of the resources comprising 
the computing package is often a significant issue within 
organizations. Directly, computers constitute a resource be¬ 
cause they bring with them differential control of their ca¬ 
pabilities vis-a-vis other resource controllers. Indirectly, 
computers raise political issues because they frequently af¬ 


fect the relative distribution of other resources such as staff, 
money and status amongst organizational units. 

Computing also raises political issues within organizations 
because of its instrumental value to different actors, partic¬ 
ularly as a result of information that can be provided. Top 
decision-makers are interested in information that might help 
clarify choices and identify problems. Managers and super¬ 
visors are interested in operational data about those whose 
activities they guide and direct. In public agencies, data- 
dependent departments like planning seek some of their data 
from information-generating departments, which sometimes 
place propietary rights on their data. Since computers fre¬ 
quently alter the content, direction, and speed of information 
flows, they thereby affect the relative influence of these 
organizational actors. 

Many problems with computing are not a function of in¬ 
adequate or unavailable technical or managerial solutions. 
Rather, they are due to differences of interest or opinion 
regarding potential impacts. Beliefs about the best solution 
vary accordingly. However, solutions to computing prob¬ 
lems which are solely technically-based or management- 
based may fail because they ignore the local politics of 
computing. Furthermore, some problems lack solutions, 
even when politics is considered. Technical and managerial 
solutions may create politically unacceptable impacts. Or, 
politically attractive solutions might not align with feasible 
technical and managerial solutions. 

These observations will be illustrated with a variety of 
data drawn from the URBIS studies. 


SERVICE PROVIDER OR SKILL BUREAUCRACY?— 
THE DATA PROCESSING FUNCTION IN LOCAL 
GOVERNMENT—James Danziger 

A straightforward interpretation of the data processing 
unit’s function is that is a provider of services to its clients. 
Increasingly, the formal information systems of many or¬ 
ganizations is centered in EDP. Data processing is respon¬ 
sible for providing the local government with a set of service 
activities involving data: record-keeping, record-searching, 
printing and calculating, record restructuring, and analysis. 
The quality of the service hinges on the effective utilization 
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of hardware, software, and staff skills to the accurate, use¬ 
ful, and timely treatment of data. In this analytic sense, the 
EDP function is similar to the typing pool (although com¬ 
puting is far more technical): both EDP and typing involve 
the skillful organization of data, to facilitate their use by 
other organizational staff. If one were to provide a normative 
theory of organizational roles relating to EDP as a service 
provider, top managers would have the primary responsi¬ 
bility for deciding what EDP should do, the EDP unit would 
provide these data handling services, and the users of EDP 
services would organize them for effective utilization. 

Alternatively, the EDP unit can be characterized as a 
“skill bureaucracy.’’ The distinguishing feature of a skill 
bureaucracy is its relative monopoly on some area of tech¬ 
nical expertise. While it does provide services, its behavior 
is driven by certain self-oriented values: (1) to preserve 
autonomy from outside control; (2) to dominate its relation¬ 
ship with its clients; (3) to expand its domain of activity; 
and (4) to be guided by its internal standards of profession¬ 
alism. There is good evidence, for example, that skill bu¬ 
reaucracies in education and public-welfare achieve these 
values relative to policy-makers and to clients of their own 
services. Given the highly technical nature of computing and 
the professional concerns of computing specialists, EDP 
units seem particularly likely to manifest attributes of a skill 
bureaucracy. 

The concepts of service-provider and skill bureaucracy 
are ideal-type characterizations, and actual EDP units dis¬ 
play some aspects of each kind of orientation. However, 
data drawn from surveys of computing practices in local 
government under the URBIS project indicates that skill- 
bureaucratic features are common in many American local 
government EDP organizations. This talk will address these 
alternative models of the EDP function based upon both 
survey data and case studies. 


USEFUL APPLICATION OF POLITICS IN 
COMPUTING—Einar Steffereud 


THE ORGANIZATIONAL ENVIRONMENT 

In many large organizations computing allocations are 
made either by committee or by a “computing czar.’’ While 
these choices seem to span the range of centralized authority 
through participatory decision-making, in fact they often fail 
to achieve the intended effects. This talk provides a frame¬ 
work for understanding how committees can be effectively 
organized to deal with different computing rationales used 
by operating units, and thus to act so as to sensibly distribute 
resources rather than simply diffuse responsibility. 

Computing has assumed a central role as a major engine 
of production and is shared by large portions of many or¬ 
ganizations. Despite the deployment of mimicomputers, 
many organizational units are becoming increasingly de¬ 
pendent upon shared or common computing systems (in¬ 


cluding networks). While it is common for different organi¬ 
zational units to develop their own rationales for action, 
shared data and computing resources tends to lead to smaller 
differences of rationale at the operating levels of large or¬ 
ganizations. 

PROBLEMS OF COMPUTER GOVERNANCE 

As computing becomes more pervasive, it becomes more 
essential to find workable compromises between value sys¬ 
tem differences between operating departments in an organ¬ 
ization. These negotiated compromises are essentially 
“political’’ rather than “technical” solutions. (Conflicting 
values cannot be forced by the authority of a computing 
“czar.” The half-life of a computing “czar” is inversely 
proportional to the extent to which he assumes control of 
other people’s resources.) 

So how can the problems of computing governance by 
effectively resolved? An effective governance structure 
must aggregate the required power from operating depart¬ 
ments. This differs from a responsibility diffusion committee 
which includes “representatives” from operating depart¬ 
ment who have little power to commit their departments to 
authoritative policies. Observations in several diverse com¬ 
puter using organizations indicate that user committees 
which fail to mirror the de facto power structure of the 
organization tend to be ineffective in dealing with conflicts 
over computing use, and fail to make “sensible” allocation 
decisions. With a power aggregation committee, substantial 
political issues can be resolved, which otherwise tend to be 
analyzed, debated, and avoided. 

This talk illustrates several different strategies of com¬ 
puting governance with examples drawn from a variety of 
computer-using organizations. 


ORGANIZATIONAL CONSIDERATIONS IN D.P. 

RESOURCE ALLOCATION—J. A. Sutton 

Data processing resources frequently are in great demand 
by “user-departments” who would like to utilize those re¬ 
sources. When the allocation of these resources is perceived 
as unfair, members of user-departments may become dis¬ 
satisfied with the data processing department. The fre¬ 
quency with which articles appear pointing out user-D.P. 
problems and suggesting solutions to these problems sug¬ 
gests dissatisfaction is widespread. 

This discussion focusses on organizational mechanisms 
intended to provide the necessary communication and re¬ 
source allocation between the Data Processing Department 
and departments wishing to utilize data processing services. 
Several mechanisms frequently proposed to facilitate inter¬ 
departmental transactions will be discussed, research results 
on their effectiveness will be reported, and recommenda¬ 
tions will be made for the management of the interdepart¬ 
mental interface. 

Mechanisms for the management of the interface between 
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D.P. and user departments generally fall into two categories: 
(1) formalization of interfaces (e.g., establishment of a D.P. 
Steering Committee), and (2) decentralization of some func¬ 
tion to the user department (e.g., location of systems ana¬ 
lysts in the user department). Mechanisms from both of 
these categories are frequently intended to “increase user 
involvement.” 

Organizational behavior theory suggests that the potential 
for conflict at the D.P.-user department interface is great 
and that substantial benefits might accrue from improved 
integration between departments. Recent research suggests 
that some of the mechanisms proposed to enhance D.P.- 
user relations have little impact (e.g., whether the D.P. 
manager reports to a department head or is himeslf a de¬ 
partment head has little impact on user satisfaction with 


D.P.). We observe that as D.P. departments grow in size, 
they exhibit a tendency to increase the number of formal 
mechanisms in place, yet evidence suggests that simply add¬ 
ing mechanisms has little effect on user satisfaction. How¬ 
ever, the perceived effectiveness of many mechanisms is 
closely related to user satisfaction with D.P. 

These results suggest that adding mechanisms intended to 
enhance the perception of “fair” resource allocation and to 
improve user satisfaction with D.P. may not serve the in¬ 
tended purpose. Rather, the results suggest that the quality 
(or perceived effectiveness) of interdepartmental integration 
mechanisms may be the critical issue. Hence, the efforts of 
the D.P. manager (and others) may be better spent at im¬ 
proving the effectiveness of existing mechanisms rather than 
putting new mechanisms in place. 




Area Director: 

Peter Freeman 
SofTech, Inc. and 
University of California at 
Irvine (on leave) 


Software development methodology 


OVERVIEW 

Beginning in the late 1960s with an interest in programming methodology, 
researchers and practitioners alike have begun to pay increased attention to the 
methods used to develop software. This interest in technique is driven by two 
major concerns: productivity and quality. 

As the cost of software has soared, both in absolute terms and as a fraction of 
the budget of many organizations, managers have realized that the productivity 
of the software development group was a critical factor. The call has gone out 
for improvement and this in turn has caused many people to take a new look at 
the methods they are using. 

The vastly expanded role of software-intensive systems in the operation of 
most organizations has also caused managers and users alike to look critically at 
the quality of the software produced. The most obvious quality factor is reliabil¬ 
ity, but there are many other factors such as security, efficiency, and maintain¬ 
ability. As with productivity, those responsible for building software have realized 
that one of the variables in building quality software is the method used. 

This increased emphasis on methodology has rapidly expanded beyond the 
original concern with programming alone. Today, we realize that the entire life- 
cycle of software—from initial need through years of repair and enhancement— 
must be considered when we start to address issues of quality and productivity. 

Matching this life-cycle awareness, we have organized five sessions for NCC 
‘78 that address methodological questions in different phases of software devel¬ 
opment: 

Software Design and Analysis includes papers that address questions of overall 
system definition and organization. 

Programming Methodology will focus on several topics that are relevant to the 
way in which we create individual programs. 

Software Verification, Validation, and Testing samples several new and im¬ 
portant concepts in the still-critical area of demonstrating system correctness. 
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Software Maintenance has been organized especially to address this phase that 
is often neglected in technical meetings. Although no papers appear in the Pro¬ 
ceedings, the panelists represent a large amount of experience with maintenance 
activity, making this an especially interesting session. 

User Experience with New Software Methods is an attempt to redress an 
imbalance inherent in technical meetings. A panel of six experts with personal 
and corporate experience using many of the new methods has been assembled. 
They will share their experience as users—not creators—of the new methods and 
discuss some of the problems encountered and benefits gained in implementing 
them. This should be an especially valuable session to attend, in addition to 
reading the position papers from the panelists that appear in the Proceedings. 

All of the paper sessions include a discussant to provide balance and perspec¬ 
tive on the presentations. We hope that these sessions will contribute to your 
understanding of new software methodologies and, in turn, help you and your 
organization to improve both the quality and productivity of your software de¬ 
velopment. 




A description scheme to aid the 
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INTRODUCTION 

Abstract data types have recently emerged as an important 
facility for the specification of sequential programs. Origi¬ 
nally defined in the language SIMULA, 1 abstract data types 
were soon recognized as useful in structured programming. 2 
Their recent refinement has taken three directions. First, 
they have been developed to support top-down design meth¬ 
ods. 3 Second, they have been extended so as to provide a 
basis for formal verification. 4 Finally, they have been used 
as the basis for a tool to aid program development 5 which 
admits rigorous, but perhaps incomplete, analysis. 6 The va¬ 
riety of these uses indicates the breadth of the benefits which 
accrue from facilities for high-level, abstract description. 

While abstract data types are convenient for the descrip¬ 
tion of a software system’s data storage components, they 
are not convenient for the succinct description of those 
system components which are more for the processing than 
the storage of data. This is particularly true when the com¬ 
ponents operate concurrently.** The major problem is that 
abstract data types are oriented towards describing com¬ 
ponents as structures of data which are operated upon via 


* This work was supported by a grant from Sycor, Inc. 

** By concurrent we mean parallelism which may be actually achieved by 
executing the system in a multiprocessing environment or which may be only 
apparent at abstract levels of system description and never achieved during 
system execution. 


procedure calls. Many components—e.g., a text editor in an 
operating system or a file system in a multiprocessor com¬ 
puting facility—are not naturally described in this manner. 

Extensions and modifications to abstract data types could 
be developed to increase their effectiveness in describing 
processing components in software systems—this has been 
done in the Gypsy system 7 and the Modula programming 
language. 8 In this paper, however, we develop a description 
scheme that retains many of the concepts of abstract data 
types but builds upon the concept of a sequential process. 
Further, the description scheme developed in this paper 
focuses upon the succinct modelling of a system’s process¬ 
ing components rather than their detailed programming. 

The modelling scheme presented here is an extension of 
a scheme for describing sequential process interaction 9 that 
was developed as a basis for software system analysis. 10 It 
has been prepared for use in the Design Realization, Eval¬ 
uation And Modelling (DREAM) system 11 which provides 
bookkeeping and analysis aid to the designers of large-scale 
software systems. The scheme is part of the DREAM Design 
Notation (DDN) which has been developed to permit the 
incremental development of a system’s design as a series of 
design description fragments, called textual units. 

In the next section, the basis for the modelling scheme is 
established by developing an abstract view of sequential 
process interaction as occurring through message inter¬ 
change. The following sections then develop the model of 
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a. subsystem, i.e., a collection of concurrent sequential pro¬ 
cesses. Subsystem interfaces and interaction are described 
first. Then a scheme for the non-procedural description of 
a subsystem’s behavior is presented. The paper concludes 
with a brief discussion of the ways in which the modelling 
scheme may be beneficially used during the development of 
large-scale software systems. 

The focus of this paper is upon DDN constructs for de¬ 
scribing collections of concurrent sequential processes. 
More complete treatment of this material and discussions of 
other aspects of DREAM may be found in References 12 
through 16. Also, the focus is upon the use of the con¬ 
structs—their syntax is covered in Reference 17. Some jus¬ 
tification for the constructs are given; others lie in the 
DREAM system’s general philosophy which is discussed in 
Reference 18. 

AN ABSTRACT VIEW OF SOFTWARE SYSTEMS 

A software system may be viewed as composed of parts, 
subsystems, which operate concurrently and asynchro¬ 
nously and which interact either through shared data objects 
or by the sending and receiving of messages. This is a natural 
way in which to view multiprocessor systems, since these 
systems actually have components which interact by mes¬ 
sage transmission. But this view is also appropriate when it 
is merely a logical one and the system actually runs in a 
uniprocessor environment. In this case, the view facilitates 
the decomposition of the system (and thereby the mastery 
of its complexity 19 ) and message transmission is used only 
to model the actual interactions. 

As an example of this view of software systems, consider 
the HEARSAY speech recognition system 20 developed at 
Camegie-Mellon University. One subsystem within the 
HEARSAY system is a data base, called the blackboard, 
which contains all the information about the utterance being 
recognized and the hypotheses which have been made as to 
its linguistic structure. The other subsystems within the 
HEARSAY system are processing components called 
knowledge sources. Each knowledge source inspects the 
information in the data base and augments or modifies it 
according to the rules which the knowledge source is pro¬ 
grammed to enforce. 

The interactions among these subsystems are as follows. 
Since it would be wasteful to have each knowledge source 
constantly inspect the data base to determine whether it 
should perform any processing, the data base is programmed 
to know which data base entries are of interest to each 
knowledge source and to send a signal to a knowledge source 
whenever one of the data base entries of interest to it 
changes value. Each knowledge source is reentrant and this 
(conceptually) means that each knowledge source has sev¬ 
eral internal servicer subsystems, each of which can perform 
the processing to be done by the knowledge source. When 
a knowledge source is signalled by the data base, one of 
these servicer subsystems is activated and it inspects the 
data base and makes any appropriate modifications. The 
interactions between a servicer and the data base consist of 


a sequence of message transmissions, with the data base 
returning messages in response to the requests sent by the 
servicer. 

THE MODELLING SCHEME 

A modelling scheme based upon this abstract view of 
software systems must have facilities for describing subsys¬ 
tems and their interactions. In this paper, we focus upon 
those subsystem interactions which occur through message 
transfer. We assume, therefore, the existence of a scheme 
for describing data objects that may be shared by a com¬ 
munity of concurrent subsystems.* Such a scheme must be 
able to describe many aspects of shared data objects, but 
for the present discussion it is only important that the 
scheme allow the specification of the possible values of the 
data objects. 

A subsystem model must describe three aspects of the 
subsystem. First, it must specify the interfaces to the sub¬ 
system not only in terms of the format of the information 
flowing through each interface but also in terms of what 
information may legally flow through each interface. Sec¬ 
ond, it must specify the legal sequences of message flow 
through the interfaces. Finally, it must relate the subsys¬ 
tem’s operation at one point in time to its operation at a 
previous point in time—i.e., it must specify the more global 
aspects of the subsystem’s operation. 

In the following sections, each of these aspects is dis¬ 
cussed and illustrated by examples which, taken together, 
comprise a description of the knowledge source subsystems 
of HEARSAY.** Since all of the knowledge source subsys¬ 
tems are similar with respect to their interactions with the 
data base, the description developed in the examples is of 
the class of knowledge source subsystems. To reflect that 
different instances of this class vary with respect to their 
details, such as the number of entries in the data base that 
are of interest to the knowledge source, we define a param¬ 
eterized class. This means that part of the definition of the 
class is a specification of qualifiers which may be assigned 
values when an instance of the class is created. Class defi¬ 
nitions and qualifiers are discussed more fully in Reference 
12 . 

SUBSYSTEM INTERFACES 

A subsystem’s interface consists of a set of ports through 
which messages may flow. Conceptually, a port is a com¬ 
munication line along which messages flow, one at a time, 
and which does not have any storage capabilities. Ports 
correspond to uni-directional communication lines and 
therefore have a direction in or out. 


* A scheme that was developed in conjunction with the work reported here 
is discussed in Reference 13. 

** The description is an approximation of the actual structure and operation 
of the knowledge sources in HEARSAY. It reflects our understanding of the 
description which appears in Reference 20, but has also been constructed so 
as to provide examples of the facilities in DDN. 
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jknowledge_sourcej : SUBSYSTEM CLASS 5 

QUALIFIERS j 

DOCUMENTATION j 

#_values is the number of data base entries 
monitored for this knowledge source} #_servers 
is the number of parallel servers in this 
knowledge source 
END DOCUMENTATION} 

#_values, #_servers 
END QUALIFIERS} 

await« ARRAY [l j :#_values] OF IN PORT; 

BUFFER SUBCOMPONENTS; 

signal OF [on_off_swltch] 

END BUFFER SUBCOMPONENTS; 

END IN PORT} 

make_request: ARRAY Ql i :#_servers]| OF OUT PORT} 
BUFFER SUBCOMPONENTS; 

request OF £data_base_operation^ 

END BUFFER SUBCOMPONENTS} 

BUFFER CONDITIONS} 
request** inspect, 
request=niodify 
END BUFFER CONDITIONS} 

END OUT PORT} 

get_answeri ARRAY i !#_servers^] OF IN PORT} 

BUFFER SUBCOMPONENTS} 

answer OF £data_base_response] 

END BUFFER SUBCOMPONENTS} 

END IN PORT; 

END SUBSYSTEM CLASS} 

Figure 1 

The messages which flow through a port are (ordered) 
sets of data objects. Each port therefore has associated with 
it a set of buffers, each able to store one data object. The 
set of buffers indicates the types of data objects which com¬ 
prise a message and the order in which the data objects are 
composed to form a message. 

The messages which may legally flow through a port are 
specified by giving buffer conditions for the port. A buffer 
condition is a predicate over the buffer data objects, indi¬ 
cating the set of legal values for the buffer data objects as 
well as the legal correlations among the values. OR’ed to¬ 
gether, the buffer conditions associated with an in-port (out- 
port) are analogous to a pre-condition (post-condition). 21 

In DDN, a port is defined by a textual unit which specifies 
the port’s name and direction and which has nested textual 
units which specify the buffers and the buffer conditions 
associated with the port. A set of ports is defined in Figure 
1 for objects in the class [knowledge_source].* If an object 
of this class were created with #_values having the value 
4 and #_servers having the value 3, then the object would 
appear as pictured in Figure 2. Notice that defining an array 
of ports implies that there is an array of buffers, one for 
each port. The condition associated with the make_request 
ports indicates that messages flowing out through these ports 
should have only the value inspect or modify. 

* It is a convention in DDN to enclose an identifier in square brackets when 
it is used to name a class. 



A port definition is comparable to a heading for a proce¬ 
dure stated in a programming language. It is similar in that 
it specifies a name by which the port (procedure) may be 
referenced and the number, type and order of the data ob¬ 
jects (parameters) in messages (parameter lists) processed 
by the internal components (procedure body). It differs be¬ 
cause ports allow only one-way communication whereas 
procedures provide two-way communication. The buffer 
conditions of DDN are similar to the entry and exit specifi¬ 
cations of Alphard 4 with the additional aspect that they are 
required to be valid whenever a message flows through the 
port. 

MESSAGE FLOW THROUGH A SUBSYSTEM 

The role that a subsystem plays within a community of 
subsystems is specified by a definition of the correlations 
among the messages flowing into and out of the subsystem. 
When this defines the injection of messages into the sub¬ 
system’s environment as a result of the reception of mes¬ 
sages, then it serves to specify the facilities provided by the 
subsystem. When it defines the reception of messages sub¬ 
sequent to the injection of messages into the environment, 
then it serves to specify the subsystem’s utilization of other 
subsystems in its environment. 

A subsystem’s message flow characteristics may primarily 
be defined in terms of sequences of message transmissions 
through the subsystem’s ports. This is analogous to defining 
a procedure in terms of a set of parameter/result pairs. More 
complex characteristics, called global characteristics, con¬ 
cern the correlations between transmissions occurring in 
different message transmission sequences. For example, in 
manipulating a stack, a pop operation must be preceded, at 
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some point in time, by a push operation. In this section, the 
concern is with the more easily specified sequential message 
transmission characteristics. The specification of global 
message flow characteristics is treated in the next section. 

Sequential message transmission characteristics are spec¬ 
ified by giving a set of programs, each of which is a model 
of a sequential process, called a control process. Each con¬ 
trol process model specifies a set of sequences of message 
flow through the subsystem. A control process is defined by 
a nondeterministic, procedural model which specifies the 
control process’ message reception and transmission activ¬ 
ity. Nondeterminism is allowed because it contributes to the 
clarity of the model, allowing succinct definition of the con¬ 
trol process. A procedural specification is also used to en¬ 
hance the clarity of the description. 

Messages flow in and out through a subsystem’s ports as 
a result of receive and send operations. For a send operation, 
a message is first composed using the values of the buffers 
associated with the port that is specified in the send opera¬ 
tion. Then, the message is sent out through the port to be 
placed in the link to which the port is attached* and thereby 
made available for reception by some subsystem. The con¬ 
trol process which invoked the operation is suspended while 
the message is constructed and placed in the link. Since 
links have infinite storage capacity, this suspension is rela¬ 
tively short and the send operation can therefore be consid¬ 
ered to be a non-blocking operation. 

When a receive operation is performed, the control proc¬ 
ess invoking the operation is suspended until a message is 
retrieved from the link to which the port specified in the 
receive operation is attached. The message is then decom¬ 
posed and distributed among the buffers associated with the 
port. Since it is possible to request a message when none is 
available, the receive operation can block the control proc¬ 
ess which invoked it for relatively long periods. 

Prior to a send operation, the subsystem will compute the 
values of the data objects which compose the message and 
place these values in the buffers. This computation may be 
lengthy and complicated, but neither its time consumption 
nor its detail are of interest in defining sequential message 


'[knowledge_source] « SUBSYSTEM CLASS’ 

listener! ARRAY [l««#_values) OF CONTROL PROCESSj 
MODEL| ITERATE 

RECEIVE await(MY_INDEX); 

END ITERATE j 
END MODELj 

END CONTROL PROCESS j 

Figure 3 


* Communication among asynchronous message transmitters and receivers 
requires a transmission controller that is able to store messages that have 
been sent but not yet received and requests for messages which have been 
lodged but not yet satisfied. In DDN, an idealized controller, called a link, is 
provided. Links hold messages and requests in (unbounded) bag data struc¬ 
tures. Thus they do not necessarily pass messages on in the order the mes¬ 
sages were sent, nor do they necessarily service requests for messages in the 
order the requests were lodged. Ports may be attached to links by a process 
described in Reference 14. 


' |knowledge_sourcej i SUBSYSTEM CUSS’ 

serviceri ARRAY [li:#_serversj OF CONTROL PROCESS; 

MODEL; ITERATE 

request(MY_INDEX) SET TO modify OR inspect; 

SEND make_request(MY INDEX); 

RECEIVE get_answer(MY_INDEX); 

END ITERATE; 

END MODEL; 

END CONTROL PROCESS; 

Figure 4 

transmission characteristics. Thus, in a control process, 
computational detail is suppressed by modelling it with a 
set-to operation which may be applied to a buffer data object 
and causes the buffer to assume a value prescribed in the 
set-to operation. 

To describe the sequential message transmission charac¬ 
teristics of [knowledge_source] subsystems, two control 
process types are needed. The first, specified in Figure 3,* 
models the consumption of messages which arrive at the 
await port from the data base. This models the subsystem’s 
operation with respect to signals sent to indicate a change 
of some data base entry of interest to the knowledge source. 
The variable MY_INDEX has a value in the range declared 
as the bounds of the array of control processes and is used 
to make each control process model distinct with respect to 
the buffers and ports to which it refers. 

The second type of control process is specified in Figure 

4. These control processes model the operation of the ser¬ 
vicers of the incoming signals, indicating that they present 
requests to the data base through one of the make_request 
ports and wait for answers to the requests through one of 
the get_answer ports. Note that nondeterminism may be 
specified in the set-to operation by giving a logical expres¬ 
sion which specifies the set of values which could possibly 
be assigned to the data object as a result of the computation 
modelled by the set-to operation. The control process struc¬ 
ture of a [knowledge-source] subsystem in which #_servers 
is 3 and #_values is 4 is graphically represented in Figure 

5. 

Each model is a control program over send and receive 
operations on ports and set-to operations upon buffers. The 
control constructs are Algol-like. There are constructs for 
definite iteration: an “ITERATE n TIMES” construct and 
a “FOR ALL i IN set_olLvalues” construct. WHILE and 
UNTIL constructs are available for indefinite iteration. 
Since many subsystems are designed to never terminate, 
there is also an ITERATE construct for infinite iteration. 
Conditional control may be specified by the usual forms of 
the IF construct. There is also a generalized CASE construct 
which allows “labelling” of the cases with logical expres¬ 
sions. All of the control constructs have a nondeterministic 
version. For example, the construct “ITERATE n OR 
MORE TIMES” may be used to indicate that the number 


* The quoted prefix attached to the control process textual unit in Figure 3 
indicates that this textual unit is intended to be an additional part of the 
definition of the [knowledge-source] subsystem class. This prefixing con¬ 
struction allows selective modification of a DREAM design description, 
thereby permitting incremental elaboration of a software system’s design. 
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of times, while known before iteration begins, can be any 
number greater than or equal to n. 


GLOBAL BEHAVIOR 

The facilities provided by a subsystem generally cannot 
be used in a totally arbitrary order. For example, the facility 
which a file system provides for opening a file must be used 
prior to the facilities for reading and writing the file. Thus 
there are correlations between what happens in one sequen¬ 
tial message transmission and what may happen in another 
sequential message transmission. 

This global behavior may be specified in terms of events, 
activities that may be observed external to the subsystem. 
Usually an event is the transmission of a message through 


’[knonledge_source3i SUBSYSTEM CLASS' 

llsteneri ARRAY [ 1: Rvalues 3 OF CONTROL PROCESS j 
MODEL| ITERATE 

heari RECEIVE await(MY_INDEX )j 
END ITERATE; 

END MODEL; 

END CONTROL PROCESS; 

'£knowledge_source]; SUBSYSTEM CLASS' 

serviceri ARRAY [li:#_serversj OF CONTROL PROCESS; 

MODEL; ITERATE 
start: NULL; 

ITERATE WHILE PERHAPS 

request(MY_INDEX) SET TO modify OR inspect; 
ask: SEND make_request(MY_INDEX); 

get: RECEIVE get_answer(MY_INDEX); 

END ITERATE; 

END ITERATE; 

END MODEL; 

END CONTROL PROCESS; 


'£knowledge_sourceJt SUBSYSTEM CLASS' 

EVENT DEFINITIONS; 

consult! SEQUENCE(ask, get), 
hear_and_d.o_something 5 

SEQUENCE(hear, start, REFEAT(consult)) 
END EVENT DEFINITIONS} 

Figure 7 


a port, and thus may be identified as the execution of a send 
or receive operation in one of the control process models. 
In general, an event may be associated with the execution 
of any one of the instructions in a control process model. 
(Since control process models are used as part of the be¬ 
havior description of a subsystem, they are known to an 
external observer.) 

Global behavior is specified by defining a set of sequences 
of events. Note that this is what was accomplished proce- 
durally via the control process models—each model defines 
a set of sequences of message transmissions where each 
sequence corresponds to an execution of the model. For the 
non-procedural specification of behavior, formal language 
theory provides several techniques since the set of events 
may be considered to be an alphabet and a behavior is then 
a language over this alphabet. 10 

Two aspects of global behavior—one being required for 
correct operation and the other corresponding to a design 
decision—need to be specified for [knowledge_source] sub¬ 
systems. First, a servicer must not interact with the data 
base until after it has been activated by a signal indicating 
that an entry in the data base has changed. Second, the 
interactions with the data base must be ordered such that 
no request from any of the servicers is lodged until the 
previous request has been answered—the servicers must 
coordinate their interactions with the data base so that they 
utilize its facilities in a subroutine fashion. 

To describe this behavior, we first define some events as 
indicated in Figure 6. The statement labels define names for 
events which correspond to the execution of the labelled 
statement. Note the use of the NULL instruction to allow 
denotation of the event “a servicer starts a sequence of 
interactions with the data base.” 

More macroscopic events are needed to conveniently de¬ 
fine the global behavior. These are defined by the textual 
unit shown in Figure 7. The REPEAT operator denotes that 
the consult event may be repeated zero or more times. The 
SEQUENCE operator denotes that the events which are its 
arguments are sequenced in the specified order. Thus, the 
hear_and_do_something event corresponds to a signal ar- 


'[knowledge_source^: SUBSYSTEM CLASS’ 

DESIRED BEHAVIOR; 

POSSIBLY ^servers CONCURRENT 

(hear_and_do_somethlng, £knowledge_source31 

hear_andjdo_something), 

MUTUALLY EXCLUSIVE 

(consult, l_knowledge_source3l consult) 

END DESIRED BEHAVIOR; 


Figure 6 


Figure 8 
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riving from the data base and the subsequent interactions of 
a servicer with the data base. 

Using these events, the global behavior may be specified 
as in Figure 8. The first part of the specification indicates 
that hear_and_do_something events may proceed in par¬ 
allel up to the limit imposed by having only #_servers ser¬ 
vicers. The second part indicates that interactions with the 
data base are mutually exclusive, i.e., ordered in time. 

This example has used only a portion of the facilities 
available in DDN for behavior specification. Events may be 
associated with other aspects of a subsystem’s operation, 
more complex relationships among events may be defined, 
relationships between events defined for different subsys¬ 
tems may be established, and events which are not associ¬ 
ated with any computational part of the system may be 
defined. A complete description of DDN facilities for be¬ 
havior specification is given in Reference 15. 


CONCLUSION 

The modelling scheme presented in this paper allows the 
abstract description of a collection of concurrent processes 
(a subsystem). Models in this scheme describe the interac¬ 
tions which may take place between a subsystem and other 
subsystems. A model therefore provides a specification for 
a subsystem which describes its behavior fn relation to its 
environment but hides the subsystem’s operational detail. 
A model consists of a definition of the subsystem’s interface, 
a procedural description of legal usage of the interface and 
a non-procedural description of the legal usage of the sub¬ 
system over time. 

Models in the scheme are rigorous, unambiguous speci¬ 
fications of the components within a software system. At 
the very least, the models serve to record the facilities which 
each component is to exhibit and the manner in which these 
facilities are to relate. The models provide, therefore, spec¬ 
ifications which may be used to guide the eventual imple¬ 
mentation of the components. 

The models may also be used in the formulation of argu¬ 
ments as to the appropriateness of a system’s design. While 
general techniques for formal verification cannot be defined, 
both simulation and analytic techniques can be used to de¬ 
rive information concerning the dynamic characteristics of 
the interactions among the subsystems. 22 The designer may 
then use this information to determine whether or not the 
specified desired behavior is actually achieved or to uncover 
situations in which the desired behavior is not achieved. 
This allows the designer to make corrections before pro¬ 
ceeding and to proceed with increased confidence in the 
validity of the design. 

We have found the modelling scheme to be convenient for 
the description of a variety of software systems. 23-27 We feel 
that it demarcates an important set of facilities which are 
required for the specification of software systems and that 
it will prove to be valuable as a basis for a variety of tools 
to aid designers of large-scale software systems. 
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INTRODUCTION 

Computer-based systems that perform sophisticated tasks in 
the face of challenging and variable environments are being 
developed in increasing numbers. Among the examples that 
can be cited are space exploration modules, highly auto¬ 
mated weapon systems, and data-base management sys¬ 
tems. The design and development of systems of this class 
involve many difficulties, and no generally accepted, suc¬ 
cessful methodology for the task is available. 

This paper proposes a contribution to such a methodology 
in the form of a model that is believed to be generally 
applicable to the construction of interactive systems. The 
model is based on the identification of key features common 
to such systems. These features will be described first. To 
illustrate the concepts involved, running examples of a large 
timesharing operating system, and a robot driving an auto¬ 
mobile in traffic will be used. Later, applications of the 
model to the design of microprocessor systems and multi¬ 
language processing systems will be discussed. 

INTERACTIVE SYSTEMS 

Interactive systems are complexes of people, hardware, 
and software, performing a set of well-defined tasks. These 
tasks are subject to, or initiated by, conditions met in the 
environment, and are carried out under general performance 
objectives. A timesharing operating system has an environ¬ 
ment consisting of user-manned terminals and other periph¬ 
eral devices, and its tasks are the standard types of user 
requests. A robot chauffeur has an environment consisting 

* The research reported here was supported by the National Science Foun¬ 
dation under grant MCS 76-07682. 

** On leave from the American University. 


of its passengers and the world outside the car, including 
roads, other cars, traffic signals, parking areas, and so on. 
Some of its tasks are starting, entering and leaving traffic 
lanes, parking, and obtaining fuel. Performance objectives 
for the robot include safety, comfort of passengers, and 
obedience to traffic regulations while for the timesharing 
system they include response time, throughput, and relia¬ 
bility. 

As these examples show, tasks are the standard compo¬ 
nents of system activity. Tasks are refined by breaking them 
down into common subtasks, which are further decomposed 
until they result in relatively low-level units called actions. 
For example, the robot chauffeur’s activity is expressed in 
terms of actions such as turning the ignition on or off, ac¬ 
celerating, steering and braking. These actions are the ele¬ 
ments of the subtasks of turning a comer, obeying a traffic 
signal, occupying a parking place, and so on. Actions in the 
case of the timesharing operating system include compiling 
a program, searching a directory, or moving a block of data 
to permanent storage. Actions require resources for their 
execution. For example, an action for the operating system 
may be programmed as a machine language subroutine, but 
a processor and some core memory must be assigned in 
order that the action can be performed. 

An interactive system may also receive assignments of 
unfamiliar tasks which must be carried out by plans —com¬ 
binations of standard tasks which are formulated during the 
operation of the system. A plan will require a decision as 
each task is completed about the next task to be undertaken. 
A plan is needed by the robot chauffeur for an auto trip to 
an unfamiliar destination, while the operating system needs 
a plan for a large job, involving a number of tasks which 
must be completed quickly with minimum disruption to 
other system users. 

An interactive system has input from, and output to, its 
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environment. Input may include tasks, as well as data about 
the conditions in the environment that are involved in exe¬ 
cuting tasks. For example, users of the timesharing system 
can input requests, while the robot chauffeur may sense 
traffic conditions. Output may modify the environment or 
provide information related to tasks. The operating system 
may output an error message; the robot chauffeur may turn 
on the automobile’s headlights. 

Finally, an interactive system must store and maintain 
access to a considerable amount of information. Some of 
this information is general, concerned with the set of tasks 
and their translation into actions the system can execute. 
Another part of this information is environmental data— 
e.g., automobile traffic conditions for the chauffeur and job 
mix for the operating system. Yet another part is information 
which must be maintained about the states of the various 
parts of the system as well as planning information. The 
former might refer to the status of a task for a particular 
user of an operating system, while the latter might refer to 
information to be used to determine how well the chauffeur 
is performing an unfamiliar task assigned to it. 

METHODOLOGY 

In constructing a complex interactive system, one must 
find adequate solutions for three major aspects of system 
organization. First, the execution of system tasks must be 
controlled. The actual course taken in completing a task 
depends on the particular conditions met in the environment; 
and it can be recognized that the control problem is a sub¬ 
stantial one. Second, the information needed by the system 
must be managed so as to be available when needed. And 
third, system resources (such as memory and processor time 
for an operating system) must be managed. 

A complex interactive system must be organized into log¬ 
ical parts, and this must be done so as to meet two conflict¬ 
ing constraints. For economy, costly resources must be 
shared by system elements wherever possible, and functions 
that are the same should not be duplicated. But for effective 
performance, the parts should be as independent as possible, 
so as to avoid unnecessary communication and to keep down 
competition for resources, both of which add to the cost of 
overhead. A good trade-off between independence and shar¬ 
ing must be found if the design is to achieve performance 
objectives at reasonable cost. 

One approach that has been used in constructing complex 
systems may be called bottom-up. Development begins with 
small elements that handle critical details of the system’s 
operation. Larger aggregates are formed using these parts, 
and the process is continued. The resulting system is struc¬ 
tured in layers, each with a particular capability; the top 
layer handles the tasks in the form that the system receives 
them. In the case of an operating system; this type of layered 
structure has been called a hierarchy of virtual machines. 1 
This type of hierarchy can also be realized by an approach 
called top-down which starts with a gross description of the 
system and calls for successive refinement until the small 
critical elements of the system are reached. 2 The difficulty 
with the former approach is that considerations of perform¬ 


ance and economy do not enter in the early stages of the 
development, and the early decisions may well make it im¬ 
possible later on to achieve performance objectives econom¬ 
ically. While this difficulty is removed in the latter approach, 
it is hard to apply unless one has a good understanding of 
how the system operates. 

Another approach to construction is to dissect the system 
into modules, which operate independently except for well- 
defined controls and intercommunication; see e.g., Parnas. 3 
The problem of the designer using this approach is to specify 
modules that require little duplication of information and 
low-level functions, and that can share resources to an ap¬ 
propriate degree. Useful guides to the choice of such mod¬ 
ules do not seem to be available. 

The proposed approach to modularization is intended to 
provide a guide or model of general applicability. Called 
structural decomposition, it derives from work on artificial 
intelligence. 4-7 It leads to breaking up the system into mod¬ 
ules that either directly execute some action, or else support 
its execution. Support may take the form of control, allo¬ 
cation of resources, or management of information. 

An important feature of the model is its separation of the 
short-range, intermediate-range, and long-range aspects of 
control. Short-range control is involved in executing and 
supporting individual actions. Intermediate control relates 
to the determination of the next action to be called when 
the outcome of the current action occurs. Long-range con¬ 
trol arises in planning how to carry out assignments of un¬ 
familiar tasks in terms of known tasks, in adapting the sys¬ 
tem to changes in the environment, or in otherwise 
modifying the system. Separation of control in this way 
reduces the average frequency of communication without 
restricting the kinds of communication that can take place, 
and can therefore lead to lower overhead. 

Handling input and output in interactive systems often 
presents technical difficulties. Thus, the model places com¬ 
munication with the environment in a separate module. 

STRUCTURAL DECOMPOSITION 

An interactive system is constructed out of numerous 
components, with well-defined interfaces. Some parts are 
hardware, others are software; they are likely to be human 
components also. Hardware is usually specific to the partic¬ 
ular type of system, but the other aspects of the modules 
that make up the system are being discussed. For the pur¬ 
poses of this exposition, it is irrelevant whether they are 
realized in software, firmware, or people. 

The proposed approach to the design of such systems is 
to break them down into elements in a particular way. Struc¬ 
tural decomposition implies that each module corresponds 
directly to some aspect of executing the task as a series of 
actions. The decomposition proceeds in stages, each stage 
refining the parts identified in the preceding one, until the 
system has been expressed as off-the-shelf elements. 

The first two stages of this decomposition are general 
enough to serve as a model for a wide class of interactive 
systems. The description of these stages will emphasize the 
role played by each part in carrying out tasks, and also the 
communications between parts. 
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At each stage, a specification for each part, the commu¬ 
nications it receives and the communications it issues is 
determined. This specification guides the further refinement 
of the part, considered as a finite-state machine. 


The first stage 

The initial decomposition of the system is into parts that 
deal respectively with the long-range, mid-range and short- 


range aspects of activity. These parts are known as the 
Executive, Supervisory, and Operating components (see 
Figure 1). 

The Executive is responsible for changes to the system, 
including both structural modifications and adjustments to 
adapt to variation in the environment. (This component usu¬ 
ally is realized at least partly by a human administrator, 
but since changes to the system as initially put together are 
inevitable, the design should make explicit provision for 
them.) The Executive plans the expression of new tasks, if 
any, in terms of simpler tasks already known to the system. 
It also handles system start-up. 

The Supervisor’s job is to translate tasks into sequences 
of actions. It does this by initiating the next action on the 
basis of the outcome of the action just completed. The Su¬ 
pervisor also maintains knowledge of the current status of 
the system’s tasks. 

The Operator handles input from and output to the envi¬ 
ronment. Also, it carries out the actions called for by the 
Supervisor, making available resources and supporting func¬ 
tions as required. It keeps information pertaining to the 
environment and to the status of resources. 



Figure 2—Communication in the model 
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Communications between these components must take 
place (see Figure 2). The Operator transmits to the Super¬ 
visor any tasks that it recognizes among the inputs from the 
environment. It also reports the outcome of each completed 
action, including any status information needed for speci¬ 
fying subsequent actions. 

The Supervisor calls on the Operator for each action as 
appropriate, and provides the parameters that particularize 
the action to the current situation. Also, it provides to the 
Executive status summaries needed for the decisions the 
Executive makes. 

The Executive supplies the Operator with start-up infor¬ 
mation, adjustments in operating rules, and structural mod¬ 
ifications, as these become appropriate. If necessary, it 
passes to the Supervisor the task knowledge for handling a 
new task. 

The second stage 

Each of the components of the first stage is itself decom¬ 
posed into parts. Among those parts are certain active ele¬ 
ments that perform special functions. Such elements are 
known as processors, and indeed they may be realized by 
microprocessors; on the other hand, their functions may be 
supplied by programs that share general processing units. 
At the level of this discussion, it does not matter which way 
the element will be implemented. Also, it is possible that a 
processor is busy when a new call for its function arises. In 
such a case, the processor is assumed to have its own queue. 

The “knowledge” or “information” used by the compo¬ 
nents has already been mentioned. The data structures that 
embody the knowledge are identified elements of the com¬ 
ponents in the second stage. 

The Supervisor consists of a task processor, an action 
sequencer, and two data structures—the task knowledge and 
the task status. When a task is received from the Executive 
or the Operator, or when the latter reports the outcome of 
an action, the task processor updates the task status and 
determines the next action to be called. The task knowledge 
is used in this process. If the new action must wait for 
attention along with others already called, the action se¬ 
quencer decides its place in the line. The task processor 
passes the results of tasks to the Executive. 

The Operator contains an inputloutput processor, a set of 
action processors, support units, and environmental data. 
The input/output processor handles inputs from the environ¬ 
ment, in the light of task status information received from 
the Supervisor, and recognizes tasks, as well as task-related 
data to be added to the store of environmental data. The 
processor also issues output to the environment as directed 
by the Supervisor. The action processors execute actions, 
using environmental data and task status information; out¬ 
comes are reported back to the Supervisor. The support 
units make their resources available to the processors, man¬ 
age the environmental data, and do low-level scheduling 
when necessary. 

The Executive has a planning unit and decision informa¬ 
tion. The planning unit can express a novel job in terms of 



tasks that the Supervisor is able to manage, and it directs 
the sequence of such tasks, calling each new one in light of 
the result obtained in the current one. In the process it uses 
its decision information, which includes measures of per¬ 
formance objectives and summaries of past experience. A 
task result may also indicate that rules in use by the Operator 
need modification, or even that some Operator element 
should be replaced, and the Executive will then issue the 
necessary change. 

The diagram in Figure 3 summarizes the structural decom¬ 
position arrived at in the second stage. Solid dots represent 
active modules, and open circles represent information ele¬ 
ments. Active modules may themselves be constructed by 
applying structural decompositions to them. 

Actions 

As mentioned earlier, the concept of action is central to 
the approach being discussed. An action is an atomic unit 
of activity that occurs in many tasks. It is executed when it 
has been supplied with the resources and information it 
requires, and it continues until it reaches some appropriate 
termination point. (This is in contrast to the notion of proc¬ 
ess, used in many software systems. A process can be in¬ 
terrupted at arbitrary points, and resumed later. The inter¬ 
ruption incurs an overhead cost in storing the process state 
and restoring it when it resumes.) 

In effect, actions are the operations in terms of which 
tasks are programmed. The choice of the set of actions for 
a system is a fundamental part of the effort of constructing 
the system. An action should take a reasonable number of 
the basic clock cycles of the system. This is because each 
action initiation involves the Supervisor, and also requires 
that resources and information be supplied; the longer the 
duration of the action, the smaller is the ratio of system 
overhead to task execution. On the other hand, no action 
should be allowed to hold on to its resources so long that 
other requirements of the system are affected adversely, and 
this condition will set a limit to the duration of any action. 

Actions should be general, so that they can respond to the 
variations in the environment. The Supervisor, when it calls 
for an action, supplies the parameters that specialize the 
action to the current situation. 

The example in Table I, representing a very small part of 
the task knowledge needed by a robot chauffeur, serves to 
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TABLE I.—Subtask: ENTER AT (entry point) AND STOP AT (type of stopping place) 


Action 

Parameters 

Outcome 

Next Action 

seek 

entry point type 

not yet found 

seek (entry point) 



found 

enter 



blocked 

(terminate) 


stopping place type 

not yet found 

seek (stopping place) 



found 

park 



not available 

(terminate) 

enter 

entry point 

not yet done 

enter 



done 

park 



accident 

(terminate) 

park 

stopping place 

not yet done 

park 



done 

(terminate) 


show how a subtask might be expressed in terms of actions; 
and how the actions are particularized by parameters. The 
subtask calls for the robot to enter some specified type of 
location—parking lot, service station, driveway and so 
forth—arrived at in a previous subtask; after entering, it is 
to park in an appropriate spot. 

The action seek requires that some place in the immediate 
environment be found that matches the type description 
given in the task knowledge. Therefore, the action is exe¬ 
cuted by the I/O module rather than some other processor. 
If the time allocated for the action is insufficient for it to 
find the place, the action seek may be continued. The sub¬ 
task is terminated when the car has been parked, or when 
completion has become impossible. In either case, a new 
subtask will be determined by the Supervisor. 


EXAMPLES 

The application of the interactive system model to the 
design of a robot chauffeur has been outlined above in gen¬ 
eral terms; for further details, see Reference 7. The model 
has also been applied to the development of a timesharing 
operating system, 8 and a brief description of this application 
will be given. 

An operating system 

The environment of the operating system, as mentioned 
earlier, are users at terminals, and other peripheral devices. 
The tasks are job requests issued by users. The environ¬ 
mental data are user program and data files—both temporary 
and permanent. Examples of actions, in addition to those 
given earlier, are running as much of a user program as can 


be completed within an allowed time quantum, and inputting 
and outputting lines of text at terminals. 

System functions are handled by the support units of the 
Operating component (called the Interface in the reference 
cited). These functions include managing core memory; al¬ 
locating I/O channels, buffers, and similar resources to ac¬ 
tions that require them; and scheduling processors to exe¬ 
cute actions. 

The Supervisor at any time is controlling the carrying out 
of each user’s current request. When it calls an action, the 
call is placed on an action queue, according to its priority, 
by an action sequencer. The action call waits until it has 
received the processor, core memory, and other resources 
needed for its execution. The action program then carries 
out the action, reports the outcome to the Supervisor, and 
releases resources that are no longer needed. The Executive 
(called the Policy component in this application) can modify 
scheduling and other system rules to adjust to workload 
changes. The system administrator, who is treated as part 
of the Executive, can plan the handling of unusual jobs. 

To illustrate the task knowledge for such a system, con¬ 
sider the subtask Open (file, access) which can be used to 
open a file. The following actions comprise the subtask: 

transfer (file part, buffer) —transfers the next portion of 
the indicated file into a buffer allocated by the system. 

search (buffer, file descriptor, access) —searches the 
buffer area to find a file descriptor. If present it deter¬ 
mines whether the user has the indicated access right. 

output (message )—prints the indicated message on user’s 
terminal. 

A representation of a simplified version of the subtask in 
terms of the above actions is described in Table II; the initial 
action is transfer. 


TABLE II.—Subtask Open (file, access) 


Current Action 

Parameters 

Outcome 

Next Action 

transfer 

next portion of directory, buffer 

transferred 

search (file, access, buffer) 

search 

file, access, buffer 

found and access okay 

output (‘OPENED’) 



found and access not okay 

output (‘ILLEGAL ACCESS’) 



not found 

transfer (next portion of directory, buffer) 



EOF 

output (‘FILE NOT PRESENT’) 

output 

message 

outputted 

— 
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A microprocessor system 

As another example of the use of the model, a micropro¬ 
cessor system will be discussed (a more general micropro¬ 
cessor system is described in Reference 9). It is believed 
that the proposed model with its attendant concept of actions 
allows a methodology for the integration of hardware and 
software in a natural way. One example of this is in the 
realization of actions—an action may be dedicated to one 
microprocessor (or specialized piece of hardware), or sev¬ 
eral similar actions may be assigned to one microprocessor. 

Consider the I/O terminal subsystem in Figure 4 that has 
been factored from a timesharing operating system in order 
to meet certain performance objectives (e.g., efficiency). 
Lines of input from CRT terminals are considered as mes¬ 
sages to be transmitted between the operating system and 
the I/O subsystem. The subsystem itself can be decomposed 


applying the model. The Supervisor for such a system would 
take signals from the terminal, considering them as tasks, 
and would translate them into actions which would be exe¬ 
cuted by the Operator component. An Executive component 
could conceivably be used to regulate I/O traffic. 

Many designs and implementations exist for such a sub¬ 
system along with their attendant opportunities for parallel¬ 
ism. A simplified hardware/software realization of the I/O 
subsystem using microprocessors is given in Figure 5. It has 
been assumed that the number of terminals is so large that 
it is infeasible to have the I/O subsystem share a processor 
with other parts of the system. Here there are only two 
components of the model—the Supervisor and the Operator. 
The Supervisor has a dedicated microprocessor since it must 
handle requests from terminals as well as outcomes of action 
executions. Each action has a dedicated microprocessor 
since many terminals may be requesting service at the same 
time. Thus at any given time each action microprocessor 
can be handling different parts of tasks issued from several 
terminals and the operating system. Other elements of the 
configuration include a decoder/router for routing action 
calls from the Supervisor to their respective action queues, 
an I/O bus and a random access memory. 

The I/O bus requires two message formats—the I/O Su¬ 
pervisor format (see Figure 6) and the random access mem¬ 
ory format. There is one sub-format for tasks and one sub- 
format for outcomes. Tasks are issued by the timesharing 
operating system or by a CRT terminal, while outcomes are 
the results of action executions. 

During regular operation, terminals signal the system by 
placing an Input task on the I/O bus in Supervisor format. 
The task is recognized and placed on the Supervisor queue. 
When the task gets to the head of the queue, the Supervisor 
issues the first action call in the action sequence that com¬ 
prises the task (see Table III). 

In this case, the first action is transfer which transfers a 
message (synchronously) from the specified terminal to an 
internal buffer. The next action check determines whether 
the message is a command or a data item. If it is a legal 
command, it is sent to the operating system for execution, 
if not transfer outputs an error message to the appropriate 
terminal. If a data item is encountered, then add makes it 
the next line of the user’s workspace. 

The Supervisor determines the next action for a task on 
the basis of the outcome of the present action for this task. 
At any point in time an action microprocessor can be proc¬ 
essing an action call obtained from the head of its action 
queue. When finished, the outcome is placed on the I/O bus 


TABLE III.—Input (terminal) 


Current Action 

Parameters 

Outcome 

Next Action 

transfer 

terminal, action 

transferred 

check (buffer) 

check 

buffer 

command 

send (buffer) 

send 

transfer 

buffer 

buffer, workspace 

illegal command 
data 

sent 

added 

overflow 

transfer (‘ILLEGAL COMMAND’, terminal) 
add (buffer, workspace) 

transfer (‘WORKSPACE OVERFLOW’, 
terminal) 
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Figure 5—A hardware/software realization 



and a new action call is obtained from the associated action 
queue. The system recognizing that the information on the 
bus is an outcome, puts it on the Supervisor queue. Even¬ 
tually the Supervisor removes this outcome from its queue, 
and examines the associated task data structure to determine 
which action call to issue next. 



Figure 6—I/O bus sub-formats 


The action transfer is a bit different from the others. It 
essentially initiates an I/O operation; when the operation is 
complete, the associated terminal places the outcome on the 
I/O bus. The Supervisor of the operating system can forward 
a task to the I/O subsystem in an analogous way by placing 
a call on the I/O bus. The outcome can be returned to the 
Supervisor of the operating system through the send action. 


A multi-language processor 

As a final example, the application of the model to the 
design of a processor which executes a user’s program writ¬ 
ten in an arbitrary combination of programming languages 
will be discussed. The model provides a natural framework 
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in which to realize such a processor. As a prelude, the utility 
of a multi-language system will be elaborated on. 

Being able to write a program in a programming language 
suitable to expressing the solution of a problem in a straight¬ 
forward manner is very desirable. String processing algo¬ 
rithms are easily written in SNOBOL, algebraic and vector 
computations are easily represented in APL, and input/out¬ 
put operations are most effectively specified in PL/I. Since 
a universal programming language which can do all of these 
things well is impractical at this time, it is desirable to be 
able to use statements from a variety of languages to express 
a given algorithm. 

The usefulness of this approach is particularly evident in 
the following program segment which contains PL/I and 
APL statements: 

IF x - 1=0 THEN Y<—H/ZjXW; 

ELSE Y<—l-/Z 2 xW; 

PUT DATA (Y); 

When Zj, Z 2 and W are conformable matrices, even this 
simple two line segment is difficult to program in either PL/ 
I or APL. It is difficult in PL/I because (1) vector operations 
are not available, (2) PL/I subroutines for APL operations 
require a different syntax, and (3) PL/I subroutines do not 
return arrays as values. It is difficult in APL because the 
equivalent of PUT DATA naming each element would re¬ 
quire the user to generate the names of each element. 

The proposed model offers a means to realize such an 
approach in a natural way. Consider an interactive time¬ 
sharing system with a subsystem able to handle interpre- 
tively user programs written in arbitrary combinations of 
programming languages. Here the operating system is used 
to create programs, store and retrieve files, provide I/O 
capabilities, etc. The subsystem is invoked as an action 
interpret applied to a user program. 

The subsystem consists of a Supervisor and an Operator. 
The Supervisor calls upon actions executed by the Operator 
which (1) provide a generalized fetch-execute cycle for sev¬ 
eral language processors, and (2) directly execute [10] state¬ 
ments in these languages. Here, then, operations such as 
add and compare are no longer the basic units of computa¬ 
tion; statements in specific languages are the basic units of 
computation. Thus, given an assignment statement 

X=A*B+C/D 

in a program file which is tagged with some language specific 
information (e.g., language type), an action sequence con¬ 


trolled by the Supervisor gets the statement from the pro¬ 
gram file, identifies the indicated language, computes the 
expression and performs the assignment. 

Variables and data structures must be specified in such a 
way that the meaning of each statement is unambiguous, 
even if it may be interpreted in more than one language. 
Each action can have more than one outcome. If an action 
associated with a statement in a particular language executes 
properly, it may, for example, assign a value to a variable 
or indicate the next statement to be executed. Otherwise, it 
may fail to parse or encounter execution difficulty; either of 
these will cause a return to the operating system. 

CONCLUSION 

This paper has presented a model and methodology for the 
construction of interactive systems. Its usefulness has been 
shown in its application to realizing timesharing operating 
systems, robot-chauffeured automobiles, microprocessor 
systems and multi-language processor systems. It is felt that 
a structural approach of this nature is necessary in managing 
the complexity of the decomposition of large interactive 
systems. 
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Software fault-tolerance in the Pluribus 
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INTRODUCTION 

Over the past decade, the decreasing cost of minicomputer 
components has encouraged the use of multiprocessor tech¬ 
niques in the design of high-speed, cost-effective computer 
systems. Multiprocessor architectures have two principal 
advantages over conventional single-processor designs. 
First, a multiprocessor system can achieve greater compu¬ 
tational speed through parallelism in its task structure. A 
second advantage is that multiprocessors realize greater re¬ 
liability through redundancy of processing elements. This 
paper discusses several aspects of multiprocessor reliability 
as it applies to the Bolt Beranek and Newman (BBN) Plu¬ 
ribus system. 1 

Recent advances in processor technology, including the 
development of inexpensive LSI-based microprocessors, 
have further increased the cost advantages of multiprocessor 
systems and have led to the consideration of multiprocessor 
architectures involving very large numbers of processors. 
Early examples of multiprocessor systems designed around 
minicomputers include the C.mmp system at Camegie-Mel- 
lon ? ’ 3 and the BBN Pluribus. Each of these early systems 
has prompted the development of larger microprocessor- 
based systems, such as the Carnegie-Mellon Cm* project 4 
and the BBN “Butterfly” machine. 

THE BBN PLURIBUS 

The BBN Pluribus system has been described in detail in 
a number of previous papers. 1,6 The Pluribus was originally 
designed as a reliable, modular, high-speed Interface Mes¬ 
sage Processor (IMP) for use with the ARPANET. 7 To date, 
eight such systems have been delivered. The Pluribus has 
also been used to support a real-time processing and display 
system for seismic data 8 and a terminal handler connecting 
a large number of terminals to several host computers. 9 
Future applications include a high-speed version of the Sat¬ 
ellite IMP. 10 On the whole, our experience with building 
Pluribus systems has been favorable, and we are currently 
trying to attract a more general market. 

The basic organization of a typical Pluribus system is 
illustrated in Figure 1. Each processor bus contains a num¬ 
ber of processing units (usually two) and associated local 
memory for each processor. The processors are SUE min¬ 


icomputers produced by the Lockheed Electronics Corpo¬ 
ration. Common memory (that which is accessible to all 
processors) is distributed among the system memory busses. 
The I/O bus units provide an interface to a variety of input/ 
output devices, such as communication lines and modems 
in conventional Pluribus applications. In addition, each I/O 
bus includes a real-time clock used to coordinate processor 
activity and a special hardware device called a PID (Pseudo 
Interrupt Device) used for process scheduling. The individ¬ 
ual busses are connected to each other via bus couplers 
which are indicated by the solid lines in Figure 1. 

The hardware organization of the Pluribus makes it pos¬ 
sible to achieve system reliability through the redundancy 
of hardware components and interconnection paths. For 
example, if a processor bus becomes unusable through hard¬ 
ware failure, the remaining processor bus(ses) will generally 
be able to provide sufficient computational power to run the 
system. Similarly, if the connection between a processor 
and common memory bus is lost, the system can proceed 
normally by removing either of the two busses from the 
operational system. 


RELIABILITY IN THE PLURIBUS 

Fault-tolerant computer techniques originally arose in re¬ 
sponse to the unreliability of early digital hardware. 11 His¬ 
torically, the principal concern of fault-tolerant system de¬ 
sign has been with completely error-free system operation. 
The classical technique used to provide reliability at this 
level is to include three or more instances of each system 
component; each component performs each calculation si¬ 
multaneously. Additional hardware is then used to check 
the individual results for correctness, relying on a majority 
decision in the case of a discrepancy. 

The notion of relaxed reliability 

Most of the past research in reliable computer system 
architecture has been directed toward applications in which 
completely fault-free operation has been necessary. In de¬ 
signing the Pluribus, we have been concerned with appli¬ 
cations whose reliability requirements are much less strin¬ 
gent. We have found that these applications suggest a 
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Figure 1—Typical Pluribus System Configuration 
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relatively new set of techniques and that many of the classi¬ 
cal approaches to fault-free reliability do not directly apply. 

In the Pluribus, we have not been as concerned with 
continual error-free operation as with providing maximum 
system availability. In the environment of a communications 
network, for example, it is desirable to keep each node 
functioning even if data is lost during a failure-induced tran¬ 
sient. A communications network is characterized by a rel¬ 
atively steady-state operation which is largely independent 
of all but the most recent network activity. If a node in the 
network is forced to reinitialize, some active message traffic 
will generally be lost, but the node will very quickly be able 
to handle new messages in a normal fashion. This property 
of the communications network makes a relaxed reliability 
mechanism very attractive in that the continuity of system 
operation can be insured without incurring the cost of a 
more elaborate hardware reliability scheme. 

We feel that the relaxed reliability model is widely appli¬ 
cable in other areas. For example, by enhancing a general- 
purpose operating system with checkpointing facilities, it 
should be possible to create an environment where such 
approaches to fault-tolerance may be used to advantage. 
The techniques used in the Pluribus should permit such a 
system to reconfigure following a failure and automatically 
resume normal operation from the previous checkpoint. In 
the network environment, checkpointing is not necessary 
since there is no long-term context to be saved. A system 
with checkpointing approximates the network environment 
by providing a clean state (the previous checkpoint) at which 
operation can resume. 


Relaxed reliability in the Pluribus 

Unlike most conventional systems, the principal respon¬ 
sibility for maintaining reliability in the Pluribus is placed on 
the system software rather than in the hardware structure. 
The Pluribus hardware was designed to provide an appro¬ 
priate vehicle for the software reliability mechanism. When 
hardware errors are detected, the software exploits the re¬ 
dundancy of the hardware by constructing a new logical 
system configuration which excludes the failing resource, 
using redundant counterparts in its place. 

Pluribus systems make use of redundancy in their soft¬ 
ware structures for much the same reason. Redundant in¬ 
formation is intentionally introduced into the data structures 
at various points and checked by processes operating upon 
those structures. An example of this technique applied to 
buffer structures is described later. In addition, periodic 
background processes are used to recompute certain varia¬ 
bles which are maintained by the operational system. If the 
recomputation uncovers a discrepancy, the variables are 
fixed or a more drastic recovery is attempted. 

In many cases, a failure is not detected at the exact time 
of occurrence but at the time that the software encounters 
some failure-induced discrepancy. By this time, the effects 
of the failure may be more widespread and the actual cause 
of the failure may be difficult to detect. In such cases, the 
system is not able to perform instantaneous recovery and 
seeks instead to restore normal operation as quickly as pos¬ 
sible. 
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The value of software reliability 

Our experience with the Pluribus has convinced us that 
fault-tolerance need not be expensive, provided complete 
fault-free operation is not the goal. Pluribus hardware can 
be configured for redundancy by including one extra copy 
of each critical system resource (i.e., one extra processor 
bus, one extra memory bus, and so forth). In general, soft¬ 
ware reliability mechanisms are highly flexible and are 
largely independent of the application routines. 

As a result of placing the responsibility for system recov¬ 
ery on the software, Pluribus systems are capable of surviv¬ 
ing (or isolating the effects of) a wide variety of intermittent 
failures and design errors in either hardware or software. 
Diagnosis of the failures can often be accomplished while 
normal operations continue. 

THE PLURIBUS OPERATING SYSTEM 

This section discusses the organization of the Pluribus 
operating system and some of the techniques used for 
achieving coordination of multiple processors. These tech¬ 
niques are further explored in later sections, which provide 
two examples of Pluribus fault-tolerant software strategies. 
One of these examines the Pluribus IMP buffer system in 
detail, and the other covers strategies for understanding 
failures when they occur and effecting necessary repairs. 

General responsibility of the operating system 

The software reliability mechanisms for a Pluribus system 
are coordinated by a small operating system (called 
“Stage”) which performs the management of the system 
configuration and the recovery functions. The overall goal 
of the operating system is to maintain a reliable, current 
map of the available hardware and software resources. The 
map must include accurate information not only about the 
hardware structure of the machine, but also about variables 
and data structures associated with the processes that use 
that hardware. Moreover, the operating system must func¬ 
tion correctly even after parts of the system hardware have 
ceased to be operational. New resources, as they are dis¬ 
covered (e.g., because hardware has been added or re¬ 
paired), should be incorporated as part of the ongoing op¬ 
eration of the application system. 

Since any component of the system may fail at any time, 
the operating system must monitor its own behavior as well 
as that of the application system. It may not assume that 
any element of hardware or software is working properly— 
each must be tested before it is used and retested periodi¬ 
cally to ensure that it continues to function correctly. The 
operating system must be skeptical of its current picture of 
the system configuration and continually check to see if the 
environment has changed. 

The Pluribus operating system builds the map of its en¬ 
vironment step by step. Each step tests and certifies the 
proper operation of some aspect of the environment, relying 


on those resources certified by previous steps as primitives. 
Early steps examine the operation of the local processor and 
its associated private resources. Subsequent steps look out¬ 
ward and begin to discover and test more global resources 
of the system, giving the checking process a layered ap¬ 
pearance. In the Pluribus operating system, each processor 
begins by checking its own operation and by finding a clock 
for use as a time base. Once these resources have been 
verified, the processor can begin to coordinate with the 
other active processors to develop an accurate picture of 
the system. 

At the same time, the system must balance the need for 
reliable primitives with the need to accomplish normal op¬ 
eration efficiently. When all the environment has been cer¬ 
tified, the system should spend most of its processing power 
on advancing the operational algorithms and return only 
occasionally to the task of re verifying its primitives. When 
failures of the environment have been detected, however, 
the power of the system must be brought to bear on the task 
of reconfiguring to isolate the failure. 

Hierarchical structure of the stage system 

The Pluribus operating system is organized as a sequence 
of stages which are pooled by a central dispatcher. A pro¬ 
cessor starts with only the first stage enabled. As each stage 
succeeds in establishing a proper map of its segment of the 
system state, it enables the next stage to run. Each stage 
may use information guaranteed by earlier stages and thus 
may run only if the previous stage has successfully com¬ 
pleted its checks. Once enabled, a stage will be polled pe¬ 
riodically to verify that the conditions for successful com¬ 
pletion of that stage continue to apply. The system applies 
most of its processing power to the last stage that is enabled 
but returns periodically to poll each earlier stage. The ap¬ 
plication system is the final stage in the sequence and may 
run only after the earlier stages have verified all the config- 


TABLE I.—Pluribus Operating System Stages 
STAGE FUNCTION 

0 Checksum local memory code (for stages 0, 1, 2). Initialize local 
interrupt vectors, and enable interrupts. Discover Processor bus 
I/O. Find some real-time clock for system timing. 

1 Discover all usable common memory pages. Establish page for 
communication between processors. 

2 Find and checksum common memory code (for stages 3, 4, 5). 
Checksum whole page (“reliability page”). 

3 Discover all common busses, PIDs, and real-time clocks. 

4 Discover all processor bus couplers and processors. 

5 Verify checksum (from stage 2) of reliability page code (for rest 
of stages plus perhaps some application routines). External re¬ 
loading of missing code pages is possible once this stage is run¬ 
ning. 

6 Checksum all of local code. 

7 Checksum common memory code. Maintain page allocation map. 

8 Discover common I/O interfaces. 

9 Poll application-dependent reliability and initialization routines. 
Periodically trigger restarts of halted processors. 

10 Application system. 
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uration information of the application and the validity of the 
data structures. 

Table I lists each stage of the Pluribus operating system, 
together with the aspects of the environment it guarantees. 
Many of the functions listed will not be discussed further 
but are provided to illustrate the layering of stages. 

Since processors continue to perform each of the stages 
periodically, changes in the environment will eventually be 
noted. Any stage detecting a discrepancy in the configura¬ 
tion map will disable all later stages until the discrepancy is 
repaired. Then, all the later stages, which might depend on 
data verified by the disabling stage, will be forced to run all 
their checks, guaranteeing that they will make any further 
modifications to the configuration map necessitated by the 
first change. A serious failure, such as a non-existent mem¬ 
ory interrupt, disables all but the first stage. In these cases, 
some reconfiguration might be needed, and all stages should 
perform all their checks before the application system is 
resumed. 

Establishing communication 

So far, we have described the progress of one processor 
through the staged checking procedures of the operating 
system. All processors in the Pluribus perform the same 
checks, since it is important that they agree about the state 
of the system resources. Coordination of multiple processors 
with potentially different views of the hardware configura¬ 
tion requires two mechanisms: the processors must agree 
on an area of common memory in which to record the 
machine configuration map, and they must cooperate in their 
decisions to modify the map. 

The first step in coordinating the multiple processors of 
a Pluribus is to agree on a page of memory through which 
to communicate. The procedure for initially establishing the 
page for communication is clearly delicate. Prior to estab¬ 
lishing the page, the processors have no way to communi¬ 
cate about where it will be. The procedure must operate 
correctly in the face of failures which might leave some of 
the processors seeing a different set of common memory 
pages from the rest. Processors which are unable to see the 
communication area will attempt to use another memory 
page and must be prevented from interfering with the un¬ 
affected processors. 

Any processor that is first starting up (or restarting after 
some massive failure) can assume nothing about the location 
of the communication page. Any page may be used, and 
therefore a small area for communication control variables 
is reserved on each page of common memory. Part of this 
area is used for a brief memory test, which must succeed 
before the page may be used at all. Every processor attempts 
to establish the lowest-numbered (lowest address in memory 
space) page that it sees as the page through which to 
communicate. To be valid, any page must have a pointer to 
the current communication page, and the communication 
page must point to itself. 

Each processor looks at the pointer on the lowest-num¬ 
bered page it can see. There are three possible states for the 


pointer. First, if it points to the page itself, the processor 
has found the communication page and may now proceed to 
interact with other processors about the common environ¬ 
ment. If it points to a higher-numbered page, the processor 
may just fix the pointer, as the requirement that the com¬ 
munication page be lowest makes this case inconsistent. If 
it points to a lower-numbered page, the processor must 
attempt to check if the indicated communication page is 
active. It must assume that the data might simply be old or 
invalid and must time it out using a dedicated entry in a 
special array of timers which is allocated on each page. The 
processor increments the timer, and, if it ever reaches a 
certain threshold, unilaterally fixes the communication 
pointer, and starts to use this page for communication. The 
processor is prevented from doing this by any other proces¬ 
sor which is successfully using the lower-numbered com¬ 
munication page; such a processor will periodically zero all 
the timers in the array for each memory page in the system 
before the threshold is reached. 

Consider what happens during various possible hardware 
failures. If the memory bus containing the communication 
page is lost, all processors will attempt to establish a new 
communication page on the other bus. Using their timers on 
the new lowest page (which initially points to the old one 
after the failure), they await the threshold. No one is holding 
the timers to zero, so the new page becomes the commu¬ 
nication page when some processor’s timer first runs out. 

A processor blinded to the communication page by a bus 
or coupler failure will try to establish a higher-numbered 
page for communication. From the point of view of the 
failing processor, this case is indistinguishable from the pre¬ 
vious case, where the common bus failed. Since the rest of 
the processors are satisfied with the communication pointer, 
they will hold all timers to zero, and the failed processor 
will never be able to change the communication page 
pointer. If the processor sees a set of pages disjoint from 
the rest of the system, it behaves as if no other processors 
are running, but there is no memory where it may interfere 
and now we have two systems operating independently. In 
this case it is likely that the two systems will interfere over 
other resources; since multiple failures are required for this 
situation to occur in a Pluribus, we choose not to attempt 
recovery here. 


The consensus mechanism 

When configuration data must be updated, it is crucial to 
coordinate the Pluribus processors before making the mod¬ 
ification. The mechanism to accomplish this goal we call 
consensus. Each stage has a consensus which is maintained 
as part of its environment. The first step in forming a con¬ 
sensus is to determine the set of processors that is executing 
the corresponding stage. This set has certified the primitives 
necessary to maintain successfully this stage’s portion of the 
configuration map. In order for the system to respond to 
failures, the consensus must be kept current—new proces¬ 
sors must be able to join it rapidly and processors that may 
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have halted or ceased to run the stage must be erased from 
the set. 

Each processor, based on its hardware address in the 
Pluribus, is assigned a bit in three consensus arrays, called 
“next,” “smoothed,” and “fix-it”. As part of running the 
corresponding stage, every processor periodically sets its 
bit in the next consensus array to show that it wishes to 
participate in the consensus. After enough time has elapsed 
for each properly rilnning processor to set its bit, this array 
is copied into the smoothed consensus and cleared. The set 
of processors in the smoothed array will then be used as a 
basis for decisions to reconfigure some portion of the re¬ 
source map. 

Any processor which wishes to modify some configuration 
information sets its bit in the appropriate fix-it array. Pro¬ 
cessors that agree with the configuration map clear their 
bits, and bits corresponding to processors not in the 
smoothed array are also cleared. 

In effect, the bits in the fix-it array represent the votes of 
the individual processors in favor of a potential modification. 
In most cases, it is desirable that all processors agree before 
making the change. All processors wait until the fix-it array 
matches the smoothed array before implementing the fix. 
Other modifications might require only majority or two- 
thirds agreement. The choice of policy often depends on 
some trade-off between resources (e.g., should we use more 
memory or more processors?). The Pluribus approach al¬ 
lows us to make this choice independently at each stage. 

Since each processor in the Pluribus performs each stage 
of the checking code, the consensus mechanism provides 
the coordination needed to change the configuration map 
gracefully. As one of its stages detects a failure, the proces¬ 
sor sets the appropriate fix-it bit and disables the following 
stages. When enough processors detect the failure they im¬ 
plement the fix to the configuration map. Now these pro¬ 
cessors can complete the later stages, devoting their atten¬ 
tion to any further changes required by the failure. A 
processor which sees a different picture of the resources 
and cannot reach agreement with the rest of the system 
hangs forever at the point of detecting the discrepancy. This 
technique effectively prevents the processor from damaging 
the system. 

Application-dependent checking 

In general, it is desirable for the application system to 
perform its own checks before initiating or resuming normal 
operation. The last stage provides a mechanism which polls 
application-oriented processes to perform consensus-driven 
checks and repairs of their own data structures. This stage 
uses the results of the hardware (application-independent) 
discovery stages to certify its own data structures. For ex¬ 
ample, it could allocate or deallocate device parameter 
blocks as the devices are discovered or disappear and ini¬ 
tialize spare memory pages for use as data buffers as they 
become available. User-written reliability checks can be per¬ 
formed on any of the application data structures, and the 
appropriate reinitialization invoked to remedy failures. 


Occasionally, it is possible for a processor checking ap¬ 
plication data structures to implement minor repairs to the 
data structures unilaterally. For major reconfigurations of 
the data structures, such as complete application system 
reinitialization, the checking routines must signal to the 
stage dispatcher that consensus is needed. The last concur¬ 
ring processor is then permitted to perform the reinitializa¬ 
tion routine. Just as the early stages guarantee the hardware 
map, the application-dependent routines have the consensus 
mechanism at their disposal to validate the system data 
structures before entering the system. In addition, the ap¬ 
plication system data structures are rechecked periodically 
during normal system operation. 

AN EXAMPLE OF APPLICATION RELIABILITY 

We use two general techniques to ensure the validity of 
data structures in the Pluribus. First, redundant information, 
where it exists, is checked for discrepancies, and appropri¬ 
ate action taken if they exist. Second, since detailed exam¬ 
ination of all data for inconsistency is deemed impossible 
for any system of non-trivial complexity, we use watch-dog 
timers to ensure the correct operation of the application 
system at various levels. As an example, we will discuss the 
buffer management strategy for the Pluribus IMP system. 

Buffers in the Pluribus IMP circulate through the system 
from queue to queue; in some cases, they may be shared 
between two or more processes. Since a compromised queue 
structure may, in general, rapidly degrade the performance 
of the system, elaborate checking methods are built into the 
IMP program at various levels. In particular, we must be 
able to detect queues that are crossed or looped and buffers 
that have been lost (are on no queue at all). 

Associated with each buffer in the system is a set of use 
bits corresponding to various processes that consume buff¬ 
ers. Any process that enqueues a buffer for some other 
process first sets the use bit for that process. When a process 
dequeues a buffer, the appropriate use bit must be on or the 
buffer will not be processed. As a special case, buffers on 
the system free list must have all their bits turned off. The 
buffer-freeing routine only returns a buffer to the free list if 
the last remaining use bit is that of the freeing process. 

This technique intentionally generates redundant infor¬ 
mation and continually validates it as a buffer circulates 
through the system. In other words, the existence of a buffer 
on a queue informs the system that some processing is 
desired for that buffer. In principle, the use bit signals the 
same thing. Each buffer-processing routine could scan all 
the buffers in the system for those with its use bit set, but 
such strategy would clearly be inefficient. The redundancy 
check gives preference to neither the queue nor the use bit 
as an indication of need for service, but rather requires 
agreement between the two indicators. When they disagree, 
the system assumes that a failure has indeed occurred and 
attempts to correct it by forcing the queue to be empty, so 
that the effects of the failure can be contained as much as 
possible. 

The use bits allow the prompt detection of looped and 
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crossed queues. In addition, an improper buffer pointer will 
often lead to a failure of the use bit check. We must also 
consider the case of a buffer which has been lost from all 
queues. This condition could arise due to a program bug or 
as a result of a queue being emptied after a use bit failure. 
We could employ a classical garbage-collection scheme for 
this purpose; unfortunately, the demand on buffers is often 
large in a high-speed communication system, and the req¬ 
uisite locking of the buffer resources during such a garbage 
collection would likely result in lost inputs. 

The recovery scheme we have chosen is a watch-dog 
timer mechanism. Each buffer has associated with it a flag 
which is set by normal activity of the buffer, which in this 
case is defined to be the periodic appearance of that buffer 
on the free list. Whenever a buffer is freed, its flag is set. In 
addition, flags for all the buffers on the free list are set 
periodically. In the high-speed communications environ¬ 
ment, where data passes through a network node very rap¬ 
idly, each buffer must appear on the free list at least once 
every two minutes. Therefore, each buffer flag is checked 
every two minutes to be sure it is set, and then cleared. A 
zero flag indicates that the buffer has dropped out of normal 
activity, and the buffer is unilaterally freed and its use bits 
cleared. In this way, any lost buffer is detected within at 
most four minutes and returned to normal usage. 

FAILURE DIAGNOSIS IN THE OPERATIONAL 

SYSTEM 

The Pluribus design permits the continuation, after per¬ 
haps a brief outage, of normal system operation following 
the failure of any single component. The component might 
be a single bit of memory or an entire common memory bus. 
Much of the responsibility for ensuring recovery from fail¬ 
ures lies in the software; very little hardware beyond that 
necessary to construct a multiprocessor should be required. 

The initial Pluribus systems performed their recovery 
quite well in many cases. Minor problems were often re¬ 
paired so effectively that the maintainers and users were 
never aware of the repair being carried out. Even following 
drastic failures, such as the loss of a common memory bus, 
normal system operation was restored within seconds. As 
we gained experience with the operational systems, how¬ 
ever, certain deficiencies in our strategies became clear. 

In some failure cases, one repair would lead to another, 
until eventually a fairly major reinitialization would be per¬ 
formed, with obvious effects on the users of the system. 
Unfortunately, the massive recovery often destroyed the 
evidence of the original failure, or occurred too late to permit 
effective diagnosis. While the stated goal of restoring the 
system to normal operation was achieved, we were left 
without any idea of why the reinitialization was necessary. 
This was particularly frustrating when the frequency of the 
reinitialization was on the order of hours or days. 

In other cases, normal operation seemed to continue, 
while some hardware failure occurred undetected. Either 
the failure was covered by effective recovery at a fairly low 


level in the system or it occurred in a redundant portion of 
the hardware which was not being exercised. A second 
failure, in conjunction with the first, would remove the last 
copy of some critical resource, causing the system to fail. 

Such experiences have led us to an improved design for 
the recovery software. Any recovery operation is reported 
by the system using a system trap (i.e., a supervisor call). 
Even in those cases where an error might occur occasionally 
in normal operation, we prefer to report the failure and filter 
out the non-important occurrences of the report. In this 
way, an incipient hardware failure is often detected early 
enough that it can be remedied well before the inevitable 
second failure brings the system down. 

We have also adopted strategies which exercise all the 
hardware in the machine periodically. Including a spare copy 
of some resource helps the system recover only if that spare 
resource is known to be good. Therefore, we force the 
system to exercise all of its resources from time to time. In 
some cases we use manual procedures, but the tendency 
has been to include automatic rotation procedures in the 
operational system software. 

One beneficial side-effect of this policy is that the opera¬ 
tional program has become the best diagnostic for the hard¬ 
ware. Traditionally, some of the most subtle hardware fail¬ 
ures occur during operation of the application system, 
though the hardware diagnostic program never detects any 
errors. By augmenting the operational system with diagnos¬ 
tic capabilities, we are able to isolate hard-to-find or inter¬ 
mittent failures, sometimes even without interrupting normal 
operation. The system-integration personnel routinely use 
the operational program as the last step in constructing a 
new Pluribus machine. 

Understanding the nature of a failure in the running sys¬ 
tem requires fairly accurate knowledge of the state of the 
machine at the instant of the failure. The initial implemen¬ 
tation of the trap mechanism recorded only the code number 
of the trap, which processor or set of processors had en¬ 
countered it, and a total occurrence count. More recently, 
we have augmented the trap mechanism to allow for saving 
a larger snapshot of the instantaneous state of the processor, 
including such information as the contents of the general 
registers, the global system time, map register settings, the 
last value read from the PID, and other important local data. 
These snapshots allow us to examine critical information 
about the failure after the fact, while permitting the recovery 
code to take effect and normal operation of the system to 
proceed. 

To aid in the development of new software, we have also 
included a password-protected facility whereby a processor 
which has detected a serious malfunction may signal all the 
other processors to stop before obtaining their next task 
from the PID. This capability allows examination of the 
failure in the global environment, which might otherwise be 
repaired by recovery code or cause some other more dra¬ 
matic failure. These mechanisms combine to provide the 
programmer with a reasonably tractable debugging environ¬ 
ment. In particular, such facilities have greatly aided soft¬ 
ware development for new Pluribus applications. 
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SUMMARY 

The structure of the Pluribus operating system has been 
described above, together with some techniques for software 
reliability and fault diagnosis. We have built Pluribus sys¬ 
tems for several applications, and our experience has been 
favorable on the whole. While the emphasis of our appli¬ 
cations has been upon communications, we expect our tech¬ 
niques could be applied to many different uses. Recoverable 
faults arise in most computer applications, and we feel that 
non-redundant computer systems would benefit from the 
incorporation of recovery-oriented operating systems such 
as that of the Pluribus. 
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INTRODUCTION 

The purpose of this paper is to propose a new computer 
based method for displaying the global structure of software 
systems. We are particularly motivated by finding a man¬ 
ageable representation of the directed graph (digraph) of 
calls within a design or an executable program, each directed 
edge representing one or more calls from a procedure to 
another. We expect such a representation to provide a better 
understanding of the relationships between various compo¬ 
nents of a software system and to lead to the design of 
systems that are better organized. 

The directed graph of calls within a program is an example 
of what we call invocation digraphs. Invocation digraphs are 
a particular class of directed graphs with at most one di¬ 
rected edge from one vertex to another or to itself. These 
invocation digraphs constantly arise during the design, im¬ 
plementation and maintenance of software systems. 

Usually, invocation digraphs are drawn by hand if they 
are drawn at all. “Structure charts” 14 provide the most 
familiar example of a graphic representation which is vis¬ 
ually informative. However, such a technique is of limited 
value in practice because of the considerable effort required 
to draw even a graph of modest size—and large graphs are 
the most interesting because they are the least understood. 
One of the major difficulties is deciding where to position 
adequate size boxes on the page so that the resulting struc¬ 
ture becomes visually informative instead of hopelessly con¬ 
fusing. Ofter, a judicious positioning of the boxes becomes 
apparent only after the first charting is essentially complete. 
Even if a good layout is obtained, the charts remain cum¬ 
bersome and expensive to maintain and modify. 

One obvious solution is to try mechanizing the whole 
drafting process. This certainly alleviates the manpower 
problem but leaves other problems still unsolved—how to 
dimension and position boxes, draw edges that do not cross 
over at random, handle the off-page connector problem and 
cope with the lack of adequate graphic devices at many 
installations. 

As an alternative, we propose an extension of the simple 
yet powerful indented representation of trees, capitalizing 
on the observation that most invocation digraphs of interest 
are loop free or almost loop free. We call this one dimen¬ 


sional, indented representation the Modular Tree Represen¬ 
tation (MTR) of the invocation digraph. 

In an MTR, the position of each node (procedure, sub¬ 
routine, . . .) is chosen systematically to reflect its relation¬ 
ship with other nodes in the structure. Although the con¬ 
nections are not explicitly materialized by lines, the specific 
location of each node in the MTR implies to the reader that 
certain connections must exist. Thus, the MTR emphasizes 
the global structural properties of the digraph while relegat¬ 
ing specific path connections to a secondary role. We believe 
this approach to be practical because invocation digraphs 
are most often used to grasp a morphological “understand¬ 
ing” of the structure of a software system rather than to 
trace through many individual paths. 


DEFINITION OF INVOCATION GRAPHS 

We start with a set of components numbered 1 through n. 
Components is a loosely defined term which will mean sub¬ 
routines, procedures, design segments, 2 . . . , depending 
upon the context. 

Let us construct the invocation graph G as follows: 

• create for each component an associated vertex, e.g., 
Vi corresponds to the /th component; 

• for all pairs of vertices (v t , v } ) draw a directed edge 
from v t to Vj if there is at least one invocation of thejth 
component from the ith one. 

In some simple cases, the invocation graph may turn out 
to be a tree but, in general, we expect a directed graph, 
some of its vertices having indegrees greater than 1. Invo¬ 
cation graphs are a subset of directed graphs since there can 
be at most one edge between any pair of vertices, distinct 
or not. Notice that invocation digraphs have loops if co¬ 
routines or recursively callable components are present. 

We shall use the nXn adjacency matrix A whose element 
A (i,j) has the value 

1 if there is a directed edge from v t to v } 

0 otherwise 
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The reachability matrix R is an nXn matrix such that its 
element R (i,j) has the value 

1 if there exists a directed path, i.e., a sequence of di¬ 
rected edges from v { to v 5 

0 otherwise 

Invocation digraphs usually have only one begin node v x , 
i.e., v x is the only vertex in G with indegree 0. But they may 
have several or even none at all if every component is at 
least invoked once. 

If several vertices with indegree 0 exist and/or the graph 
is not at least weakly connected with begin node v x (i.e., 
the corresponding undirected graph is not connected), we 
create another vertex v and draw a directed edge from v 0 to 
each node with indegree 0. We then determine if all vertices 
are reachable. If some vertices are still not reachable from 
v 0 , we select any such vertex, connect v 0 to it with a di¬ 
rected edge and compute the reachability again. This process 
repeats until all vertices are reachable from v 0 . Although 
this algorithm clearly forces the graph to become weakly 
connected (which is our requirement), it is inadequate in 
practice. In the appendix, we shall present some refinement 
to this algorithm guaranteeing that the number of vertices 
artificially connected to v 0 is minimal. At any rate, we can 
now renumber the vertices so that v 0 becomes v t and so 
forth. Thus, we have obtained a weakly connected invoca¬ 
tion graph with begin node v x . Without loss of generality, 
we can, therefore, turn our attention to weakly connected 
invocation digraphs with begin node v x only. 


MTR PROPERTIES 

The MTR is an extension of the common indented rep¬ 
resentation of trees. By design, the MTR becomes the preor¬ 
der representation if applied to a tree. In that case, the tree 
is traversed from left to right, visiting descendants first. 
Each vertex traversed is listed on a new line, horizontally 
indented to show its distance from the root. An example is 
shown in Figure 1. 

The purpose of the square bracket to the left of v x is to 
make the scope of v x visible, i.e., to identify the set of 
vertices that can only be reached by traversing v x . Since v x 
is the root of the tree, every vertex is enclosed in the out¬ 
ermost and only bracket. The role of brackets will become 
obvious when we examine more complex MTR examples 
later. 

In practice, invocation digraphs are rarely as simple as 
trees. As soon as a procedure is called from two distinct 
procedures, the invocation graph no longer is a tree. Due to 
the non-recursive nature of most software systems being 
built, we expect invocation graphs to be acyclic or almost 
acyclic. 

This observation is crucial for the MTR. Notice that it is 
simply a paraphrase of the fact that most software systems 
are hierarchical in nature. 3 They have a top and a bottom— 
high level components and low level ones. Thus, we can 


expect a representation that displays these hierarchical 
properties to be useful both at the global and local levels. 

To introduce the MTR of a non-trivial digraph, let us 
informally examine Figure 2. Looking at part 1 of Figure 2 
first, we have separated the subgraph into four disjoint trees 
by breaking the incoming edges to vertices with indegrees 
greater than 1. 

The four subtrees rooted at vertices 1, 6, 9 and 11 each 
have an associated bracket in the MTR. These brackets 
allow the following “understanding” of the graph: 

—1 calls 2, 3, and 4 which are called by no one else. 

2 calls 5 and 6, but 6 is called elsewhere 

—the trees rooted at 6 and 9 are referred to by the tree 
rooted at 1 (because they are in the broken part of the 
bracket associated with 1) 

—the tree rooted at 9 is referred to by the tree rooted at 
1 but also by the tree rooted at 6 (since it is in the 
broken part of the bracket associated with 1 but also 
indented to the right of 6) 

—the tree rooted at 13 is only referred to by the tree 
rooted at 9 (since it is in the broken part of the bracket 
associated with 9) 

In particular, we can infer from the MTR that the tree 
rooted at 13 is not known outside the “scope” of 9. Simi¬ 
larly, the trees rooted at 6 and 9 are not known outside the 
“scope” of 1. This property of not being known outside the 
“scope” of a particular vertex is fundamental for using the 
MTR. Each square bracket shields the vertices that are 
defined within it, except the root, from outside references, 
i.e., there are no incoming edges except possibly to the root. 
On the other hand, references from within the square 
bracket to outside vertices are freely permitted. 




In order to arrive at this MTR, we have made use of the 
dominance concept. It can be summarized as follows: in a 
weakly connected graph with a begin node of v l9 we say 
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that Vi dominates Vj if v^v, and every path from the begin 
node Ui to v 5 contains v t . 

We say that v t directly dominates v 3 if: 

(i) Vi dominates v } \ 

(ii) if v k dominates v } and v k i=Vi, then v k dominates v t . 

One of the important properties of the dominance rela¬ 
tionship is that every node except the begin node (which has 
no dominators), has a unique direct dominator. 

In the example shown in Figure 2, the subtrees rooted at 
6 and 9 are located within the scope of 1 because vertices 
6 and 9 have 1 as their common direct dominator. Simi¬ 
larly, the subtree rooted at 13 is located within the scope of 
9 because 9 is the direct dominator of 11 whereas 1 is only 
a dominator of 11. 

By placing the definition of each subtree within the scope 
of the direct dominator of its root, we guarantee that each 
subtree is enclosed within the most restrictive scope possi¬ 
ble. This rule provides much of the power obtainable from 
the MTR. 

In Figure 2, we saw examples of square brackets drawn 
with either a solid or broken line and defining two regions: 

—a solid region containing the subtree attached to the 
root Vi and obtained by only traversing vertices with 
indegree less than two (i.e., with at most one predeces¬ 
sor). Notice that the vertices enclosed in angle brackets 
( ) seem to violate that rule but these are only visited, 
not traversed. We call the solid region associated with 
v { the primary scope of v { . 

—a broken region containing the subgraphs whose roots 
all have v t as their direct dominator and indegree at 
least equal to 2. We call this broken region the second¬ 
ary scope of Vi . In general, there may be not just one 
but several secondary scopes of a given vertex. Two 


subgraphs are in distinct secondary scopes of v t if they 
are disconnected within the scope of v { . 



1 - 

I 

I 

I 



primary scope 

1st secondary scope 
2nd secondary scope 


nth secondary scope 


Within one of the secondary scopes of a given vertex, we 
expect to have p bracketed subgraphs, 0. Let us desig¬ 
nate by , s 2 , . . . , s p the roots of these p subgraphs and let 
L(s t ) be the indentation level of s { . By construction, we 
require that: 

L(si)>L(s s ) if R(5 i ,5 i )=lAR(5 j ,5 i )=0 
L(si)=L(sj) if R(s i ,s j ) = lAR(s j ,s i )=\ 

To make the solution practical, we also require that 
ma Xi(L(Si)) be minimum. 

This indentation strategy has the advantage that for an 
acyclic subgraph, the root of each subtree is further indented 
than the root of any subtree that references it. Notice that 
if two roots s t and s } are at the same indentations level, then 


(R(Si, Sj )=0AR(s j . Si )= 0)\/ (R (si ,Sj)= lAR(s it s t )=l) 

i.e., either they call each other recursively, possibly via 
some indirect paths, or neither calls the other. As we men¬ 
tioned earlier, most invocation digraphs of interest are 
acyclic or almost acyclic so that we expect most subgraphs 
that end up at the same indentation level not to reference 
each other. 

Whether or not there are any recursive invocations can 
be quickly determined by scanning the definition/reference 
lists in the right hand margin of an MTR. Whenever a vertex 
with indegree greater than 1 is referenced (i.e., vertices in 
( )), we append the MTR line number at which its definition 
(its subtree of descendants), is located. Conversely, when¬ 
ever a vertex with indegree greater than 1 is defined, we 
append the list of MTR lines at which it is referenced. If a 
vertex with indegree greater than 1 happens to be on a cycle, 
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Figure 2—MTR of an acyclic graph 


a **’ precedes any MTR line number at which it is either 
defined or referenced. 

Figure 3 shows an example of MTR containing recursive 
invocations. The reference lists contain **’ to indicate re¬ 
cursive paths and ‘|’ to separate references located within 
the primary scope from those in the secondary scope. For 
instance: 

REF AT (19| 72,228) 

primary secondary 
scope scope 
references references 


Finally, each reference line can be supplemented by a left 
arrow showing which secondary scope contains the corre¬ 
sponding definition. An example is shown on Figure 4. 


CONCLUSION 

We have introduced the idea of Modular Tree Representa¬ 
tion (MTR) of invocation digraphs. The algorithms neces¬ 
sary to build MTR’s in practice are presented in Appendix 
A. The MTR provides a canonical representation for invo¬ 
cation digraphs. 
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FMT_SPECIFICATION /P 60/ 
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AT 
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270 
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1 

<F M T_SP r CIFICATION /P 60/ > 




DEF 
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*269 

771 
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1 

REP£AT_SPEC /P 41/ 







272 
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<N0M7ER0_UNS!GNED_INT_CONSTANT 

/P 

105/ 
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DEF 

AT 

383 

273 
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/p 62/ 
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1 
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<NON?ERO_UNSIGNED_INT_CONSTANT 

/P 

105/ 

> 

DEF 

AT 

383 

275 
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1 

E 

/P 63/ 







276 
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<NON7ERO_UNSIGNED_INT_CONSTANT 

/P 

105/ 

> 

DEF 

AT 

383 

277 

4 


1 


1 

N 

/P 64/ 
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<NON7EPO_UNSIGNED_INT_CONSTANT 

/P 

105/ 

> 

DEF 

AT 

383 

279 

6 


1 


1 

C 

/P 65/ 







280 
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<NON7ERO_UNSIGNED_INT_CONSTANT 

/P 

105/ 

> 

DEF 

AT 

383 

281 
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1 


1 

0 

/P 66/ 
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<l»NSTGMED_INT_CONSTANT /P 104/ 

> 



DEF 

AT 

391 

?«3 
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1 


1 

M 

/P 67/ 
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<*JNS I GNEn_I NT—CONST ANT /P 104/ 

> 



DEF 

AT 

391 

785 
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1 
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/P 68/ 
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NAN7ERO_INTEGER_CONSTANT /P 107/ 






287 
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<SIGN /P 117/ > 




DEF 

AT 

404 

288 



1 


1 


<DTGIT /P 118/ > 




DEF 

AT 

406 

289 

4 


1 


1 

' K 

/P 69/ 
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1 
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<INTFGER_C0N8TANT /P 106/ > 




DEF 

AT 

393 

791 

4 


1 


1 

M 

/p 70/ 







29? 

7 


1 


1 


PP0CES50R_CH4RACTER /P 114/ 







793 



1 


1 


<APOSTROPHE /P 115/ > 




DEF 

AT 

396 

294 



1 


1 


<NONAPOSTROPHE_CHARACTER /p 

116/ > 


DEF 

AT 

397 


205 3 

296 

297 

298 


I STATEMENT_FUNCTION_STATEMENT /P 71/ 
I EXPRESSION /P 75/ > 

I <VARIARLE_NAME /P 93/ > 

I <EUNCTION_NAME /P 100/ > 


REF AT (13,28.43,54 I) 
DEF AT 307 
DEF AT 362 
DEF AT 368 


299 *5 

300 

301 


♦-— 

I LEN_SPECIFICATION /P 22/ 

I <INT_CONSTANT_EXPP /P 81/ > 

I <NONZERO_UNSIONED_INT_CONSTANT /P 105/ > 


REF AT (19 I 72.228) 
DEF AT 358 
DEF AT 383 


302 8 

303 

304 

305 
308 


I FUNCTION_REFFRENCE 

I <E XP9ESSI ON /P 

| <ARR4Y_NAMF /P 

I <PROCFOURE_NAME 

I <FliNCTION_NAMF 


/P 74/ 

75/ > 

94/ > 

/P 98/ > 
/P 100/ > 


REF AT (I 199.*312.*319. 

•326.*334 ) 

DEF AT 307 
DEF AT 364 
DEF AT 366 
DEF AT 368 


308 

309 

310 


EXPRESSION /P 75/ 

<ARITH M ETIC_FXPRESS ION /P 77/ > 
<CHARACTER_£xP°ESSI ON /P 84/ > 
<L06ICAL_EXPPESSION /P 86/ > 


REF AT (197.177,196.296. 
303) 

DEF AT 311 
DEF AT 325 
DEF AT 333 


311 8 

312 

313 

314 

315 

316 

317 


APITHMETIC.EXPRESSION /P 77/ 

<FUNf.TTON_REFERE*'CE /P 74/ > 
<ARITHMETIC_EXPRESSION /P 77/ > 

<ARRAY_ELE^EnT_NAME /P 90/ > 
<CONSTANT_MAME /P 92/ > 

<VARIAPLE_NAME /P 93/ > 

<UNSIPNED_APTTHHETIC_CONSTANT /P 103/ > 


REF AT (1200,308,*313. 

*320,*342 ) 

DEF AT *302 
DEF AT *311 
DEF AT 345 
DEF AT 389 
DEF AT 362 
DEF AT 370 


31» 8 

319 

320 


+—- 

I INTEGER-EXPR /P 78/ 

I 

| <FUNCTION_REFERENCF /P 74/ > 

| <ARITHMETIC_EXPRESSION /P 77/ > 


REF AT (1106.144.183. 

187,206,*346,*349 
DEF AT *302 
DEF AT *311 


Figure 3—Partial MTR of the FORTRAN 77 syntax 
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32 

16 


1 < 


<14> 

DEF 
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1 13 


< 14 > 


+- 

I 14 


REF AT (5,10 I) 

DEF AT 32 
DEF AT 22 
DEF AT 32 


REF AT (12 I 

DEF AT 30 
DEF AT 32 

DEF AT 30 
DEF AT 32 
DEF AT 32 


REF AT (24,27 |) 
DEF AT 32 


REF AT (4,6,8,14,15,16 
19,21,25,28,29, 
31) 


Figure 4 —MTR with arrows 
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In order to experiment with the MTR concept on real 
projects we have built an omnibus MTR processor. With 
appropriate input/output interfaces, it can be used to process 
invocation graphs generated from PDL designs, HOL source 
code, load modules. . . . We have already used the proces¬ 
sor in the following contexts: 

—as an extension of the PDL 2 processor to help under¬ 
stand the morphology of a software design. The invo¬ 
cation graph is, in that case, the graph of segment ref¬ 
erences in a PDL design. 

—as an aid in packaging software, particularly when 
grouping low level design segments into modules. The 
MTR provides an intrinsic decomposition into modules 
from which the actual module structure can be selected. 

—as an analyzer of existing packages to help understand 
the hierarchy of calls to procedures, references to ex¬ 
ternal structures. . . . An executable load module can 
be processed automatically to yield the MTR of all 
external references. 

—as an aid in designing overlay structures by applying 
the MTR to the nonoverlay load module. 

Based on our experiments so far, we can make the follow¬ 
ing observations: 

• MTR’s should be used during software development, 
particularly during the design phase. We have found 
that even a cursory examination of the computer gen¬ 
erated MTR of a design could trigger design changes 
aimed at making the resulting MTR more “structured” 
i.e., more simply nested. For instance, realizing that a 
“GET NEXT CHARACTER” design segment was not 
nested within “GET TOKEN” simply because of an 
early initializing reference, suggested making “GET 
NEXT CHARACTER” a self-initializing module. 

• MTR’s can be effectively used during maintenance to 
help assess the impact of a change on an existing pack¬ 
age. In that case, the MTR processor is applied to the 
executable load module directly. 

• Practical MTR’s can only be computer generated. This 
is due to the size of practical applications (the load 
module of the MTR processor itself has 90 vertices and 
almost 600 edges) and also to the high sensitivity of the 
MTR to small changes in the input graph. 

• MTR’s should only be computer maintained. This can 
be easily achieved because MTR’s can be produced 
cheaply (for instance, the MTR of the MTR processor 
only takes 8.85 sec of computer time on an IBM 370/ 
158 and a few thousand lines of print). Therefore, a new 
MTR can be produced every time a design change takes 
place or a new load module is generated. 

There are several questions about the MTR which have 
not been answered yet. The first important question is to 
determine user acceptance of such a tool. Admittedly, an 
MTR is not as intuitively obvious as a graphical structure 
chart. It takes a little practice to learn how to read an MTR, 
i.e., to visualize the graph through the MTR and be able to 


use the MTR to reason about the graph. The second question 
is to investigate modifications and/or extensions which might 
make the MTR even more useful in practical situations. 
These may involve modifications of the display layout or 
extensions of the MTR capabilities. For instance, one might 
try visualizing the implicit scope of data items by indicating 
the direct dominator which controls all references to a given 
data item. Then, there is the fascinating possibility of apply¬ 
ing the MTR to nonconventional areas. For instance, Figure 
3 showed the partial MTR of the FORTRAN 77 syntax. 

Clearly, further experimentation is needed to answer 
those questions and assess the full potential of the MTR 
technique. 
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APPENDIX 

Al Construction of the MTR representation 

This section deals with the algorithms required for gen¬ 
erating MTR’s. The construction of the MTR of an invoca¬ 
tion graph involves primarily: 

• finding the direct dominator of every vertex in G; 

• performing an extended topological sorting of a digraph, 
acyclic or not; 

• selecting entry points to force the invocation digraph to 
become weakly connected. 

These algorithms are either classical or extension of clas¬ 
sical ones. 
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It is important to notice that there exist several alternative the more familiar, stack oriented, depth first search algo- 
formulations for most of the algorithms described here, de- rithm. 
pending upon the choice of data structure representation. 

For the MTR, we have chosen to use bit vectors and bit 

matrices rather than linked lists for representing the graphs A2 Computation of Direct Dominators 
and their ancillary data structures. Consequently, these al¬ 
gorithms are oriented towards parallel bit vector operations. The algorithm for the computation of direct dominators 

For instance, for inding the descendants of a particular graph can be found in Reference 1, for instance. Using PDL, 2 the 

node, we rely on a bit vector “fusion” algorithm instead of algorithm can be informally written as follows: 

• COMPUTE DIRECT DOMINATORS (DIRECT_DOM()) 

LET THE VERTICES BE NUMBERED 1,2,. . . ,N WITH 1 THE BEGIN NODE 
INITIALIZE DIRECT__DOM (I)=l, 1 = 1,2,. . . ,N 
DO FOR EACH VERTEX I EXCEPT 1 

• DELETE TEMPORARILY VERTEX (I) 

I BUILD LIST OF DESCENDANTS (D()) REACHABLE FROM VERTEX (1) 

• DO FOR EVERY VERTEX J IN D() 

: : IF DIRECT_DOM(I)=DIRECT_DOM(J) 

: : SET DIRECT_DOM(J)=I 

• • endif 

• ENDDO FOR 

: RESTORE VERTEX (I) 

ENDDO FOR 


A3 Extended Topological Sorting 

We now examine the algorithms required to determine the 
indentation level of a group of p shared subtrees having the 
same direct dominator. A directed graph G' with vertices 
{v\,v' 2 ,. . . , v' p } is built as an invocation digraph where the 
p components would be the p shared subtrees. In other 
words, there is an edge between v and v ' s if and only if 
the root of the jth subtree is referenced from within the 2 th 
subtree. Notice that G' is not in general weakly connected, 
but this is not required by the algorithm that follows. 

The problem is to partition the set of vertices 
{v'i,v' 2 ,. . . ,v' p } into the smallest number of disjoint sub¬ 
sets s lt s 2 , . . . , s k such that 

(a) if there is a path from to v 5 but not from Vj to v t 
then 

v'ies v , v'jesp 1 

• TOPOLOGICAL SORTING (ACYCLIC GRAPH) 

LET THE VERTICES BE NUMBERED 1,2,. . . ,N 
INITIALLY SET RANK(I) = °°, 1=1,2,. . . ,N 
DO WHILE THERE ARE VERTICES WHOSE RANK( )=oo STILL 
; DO FOR EACH VERTEX I 
: : IF RANK(I)=0 THEN 

'• : COMPUTE PRED(), THE SET OF IMMEDIATE PREDECESSORS OF I, EXCLUDING I 

: : SET MXRNK=MAX(RANK(PRED( 1)) ,RANK(PRED(2)), . . .) 

I j IF RANK^oo THEN 

j * RANK(I)=MX RNK+1 

: : ENDIF 

: ; ENDIF 

• ENDDO FOR 
ENDDO WHILE 


(b) if there is a path both from v t to v } and v$ to v { then 
v'i, v' s es v \<v<k 

If v'i€s v , the rank of the 2 th vertex is said to be v. In the 
acyclic case, this problem is a simple topological sorting and 
we know 9 that condition (a) above is satisfied after the sort 
is performed. But if G' contains cycles, the classical topo¬ 
logical sorting algorithm cannot be applied. We shall, there¬ 
fore, extend it to handle cyclic graphs as well. 

In the simple topological sort of an acyclic graph, vertices 
and their outgoing edges are deleted immediately after they 
have been processed, and one of the remaining vertices with 
zero indegree is selected as the next vertex to be processed. 
It is equivalent but computationally simpler to replace the 
deletion operation by the assignment to each vertex of a 
rank, initially 0. Thus, the search for a vertex with 0 indegree 
becomes the search for a vertex whose predecessors all have 
a non-zero rank. 
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To handle graphs with cycles, we modify the search strat¬ 
egy so that if the search for a vertex whose predecessors all 
have a non-zero rank fails, we then search for a maximal 
circuit whose predecessors, excluding nodes on the circuit, 
all have a non-zero rank. A circuit is maximal if it is not 
possible to find another circuit that includes at least one of 
its vertices as well as some other vertex that it does not 
already contain. 

The algorithm can be carried out in two steps: 

—replace each maximal circuit by a single substitute ver¬ 
tex having the same immediate predecessor and im¬ 


mediate successor vertices, thus obtaining an acyclic 
graph but with self loops; 

—apply classical topological sorting to the derived acyclic 
graph, reflecting the rank assignment to every vertex of 
a maximal circuit when its substitute vertex is assigned 
a rank. 

This approach provides both a proof and a computational 
method. 

In order to find the maximal circuits we can use the 
following method: 

We start with the simple adjacency matrix A= (a„) defined 


I 

I 


+ 


617 

2 -> 

I 


FRdMON(IHCFRRM) /P 23/ 


M P 

2 -> 

1 


FRRTRA (IHCF.TRCH) /P 24/ 


M9 

2 -> 

1 


FDJOCS#(IHCECOMH) /P 32/ 


6 ? 0 

-> 

1 


<IHCC0MH2 /P 42/ > 



6?) 

2 -> 

1 


THCEFIDS /P 44/ 



6?? 


1 


<IRC0M#(IHCECOMH) 

/P 

41/ > 

6?3 
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<IHCEPRM /P 47/ > 



6?4 

3 
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THCFI0S2 /P 50/ 
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3 
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/P 
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/P 
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/P 
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1 
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1 
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/P 

55/ 
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3 

1 
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/P 
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1 
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4 
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1 

FCVAOUTP(IHCFCVTH) 
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4 
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/P 27/ 

649 
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Figure 5—Partial MTR of an IBM 370 load module 
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DEF AT 641 

DEF AT 669 
DEF AT 662 


DEF AT 660 
DEF AT 661 
DEF AT 669 
DEF AT 662 
DEF AT 667 


DEF AT 660 
DEF AT 668 
DFF AT 669 

DEF AT 669 
DEF AT 662 


REF AT (620 I *654) 
DEF AT 669 

DEF AT 660 
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by 

a u = 1 if there exists one or more directed edges from 
vertex v\ to vertex v' h i# j 
a y=0 otherwise 

and proceed iteratively to obtain the reachability matrix R. 

Let a lkl a lk2 . . . a lkm be the only m non-zero values of 
the first row of A. We set 

au = a kl h/a ki h/• • 7=1,2,. . . ,n 

If the new first row of A differs from the old one, we perform 
the operation again, using the m'-m new non-zero entries 
in the first row. If the new first row of A is identical to the 
previous one, the first row of/? has been obtained. We now 
proceed to obtain the second row, then the third. . . . This 
process is often known as fusion. 

When all n rows have been computed, the non-zero di¬ 
agonal entries of R indicate those vertices that belong to 
some directed circuits. If we now compute 

R=RAR T 


where the logical product is taken element by element, the 


resulting R matrix has the following structure: 


if r u = 1 
if r i} = 1 


vertex i is on a directed circuit 

then* r ii = 1 by construction, v' t and v'j are on 

the same maximal circuit. 


A4 Selecting entry points 

The method for finding descendants that we have just 
described can now be used to solve a problem deferred 
earlier, namely, the appropriate selection of vertices to make 
the graph weakly connected. Such a selection is necessary 
when the initial graph still has more than one compo¬ 
nent after all vertices with indegree 0 have been linked 
to the artificially created begin node v 0 . Let us assume 
that there are p such remaining unreachable vertices 
V={v h , Vi 2 , . . . , r ip }.The problem is to find the minimal 
number k of vertices from V sufficient to cover all vertices 
in V; i.e., all vertices in V are reachable from 1 or more of 
the k selected “entry” vertices. Notice that the solution is, 
in general, not unique since any vertex on a circuit may be 
replaced by any other on the same circuit. 

The following algorithm finds the entry vertices: 


LET E(), R(), C() BE P ELEMENT VECTORS WITH VALUE 0, 1 
INITIALLY SET E( )=0, C( )=0 
DO WHILE THERE EXISTS I SUCH THAT C(I)=0 
FIND THE DESCENDANTS (R()) OF VERTEX (I) 

I.E. R(J)=1 IF VERTEX J IS REACHABLE FROM I 
SET E( )=(E( )A7R()) 

SET E(I) = 1 

SET C( )=C( )VE( )VR() 

ENDDO WHILE 

. . . THE ENTRY VERTICES ARE I SUCH THAT E(I)=1, 1=1, . . . , p 


The selection of entry vertices is illustrated on Figure 5. 
The arrows in the left margin indicate addresses of the 
IBM OS/VS FORTRAN run time library which are not di¬ 
rectly referenced within the load module. 
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INTRODUCTION 

Since the first efforts of Farr and Zagorski 8 to those of 
Nelson 14 with the System Development Corporation and the 
more current, on-going work of Walston and Felix of IBM, 22 
there has been the desire to develop a method of predicting 
the ultimate size of a program before the programming effort 
begins. 

A technique for predicting the number of source language 
instructions necessary to meet a set of program specifica¬ 
tions could be of value to decision-makers in facilities which 
allow programmers to store source language versions of 
programs on disk for online debugging and revision. Man¬ 
agement could then estimate the amount of disk storage area 
necessary to support the development of each new program. 

Management personnel would also find it valuable to be 
able to predict the amount of object code a specific set of 
program specifications will require for differing reasons. The 
basic question of whether the program will fit in main mem¬ 
ory would be answerable for those operating in a constrained 
environment. For those utilizing partitioned memory, the 
impact upon the workload in each partition would be pre¬ 
dictable. For facilities operating in a virtual storage type 
environment, the ability to predict the number of pages a 
program will require would be of significant value in deter¬ 
mining if current disk storage capacity would be adequate 
to support projected programs. 

THE PROGRAMMING ENVIRONMENT 

The model of the programming environment used in this 
study is the one successfully utilized by this researcher to 
investigate the area of programmer productivity, shown in 
Figure 1. The rationale for this approach is straightforward. 
A set of program and programmer characteristics have pre¬ 
viously been analyzed regarding their impact on program 
development time. Most of the program processing charac¬ 
teristics and all of the programmer experience characteris¬ 
tics hypothesized as being important were found to have a 
significant effect on program development time. 4 If one ac¬ 
cepts the premise that longer program development time is 
related to larger programs, then it seems reasonable to as¬ 


sume that the variables found related to program develop¬ 
ment time should, more likely than not, also be related to 
program size. 


PROGRAM AND PROGRAMMER CHARACTERISTICS 

The purpose of this research was to determine the effect 
of program and programmer characteristics on program size, 
both in terms of source language lines of code and object 
language memory storage measured in bytes. As a conse¬ 
quence, all the remaining sources of variance in the pro¬ 
gramming environment had to be held constant. The re¬ 
search setting used was one where programmers coded off¬ 
line and only COBOL programs were considered. Since this 
is the most typical business programming setting, it was 
believed the findings of the research would then have more 
general applicability. Since the study was conducted in a 
specific data processing facility within one firm, it was felt 
that the hardware and organizational variables were then 
adequately controlled for. Only the programming problem 
and the programmer, therefore, remained as sources of var¬ 
iance in program size. 

In order to develop a reliable technique for predicting 
program size, the following questions must be answered: 

1. What characteristics of the programming task, which 
can be objectively measured before the task is begun, 
are significantly correlated with program size? 

2. What characteristics of the programmer, which can be 
objectively measured before the task is begun, are sig¬ 
nificantly correlated with program size? 

When considering the characteristics of processing re¬ 
quirements of a program which one could reasonably expect 
to have an impact on the size of the resulting ^program, the 
same variables examined for possible impact on program 
development time in previous research 4 were believed wor¬ 
thy of analysis. Those program characteristics are: 

1. Total number of input files 

2. Total number of types of input records 

3. Total number of unique fields of input records 
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PROGRAMMING MODE 
CHARACTERISTICS 

Offline Code Entry 



ORGANIZATIONAL OPERATIONS 
CHARACTERISTICS 

Programming Methods 
^Standards 

System Documentation 
Standards 


COMPUTER HARDWARE 
CHARACTERISTICS- 

Memory Size 
Storage Devices 


PROGRAMMER PRODUCTIVITY 


PROGRAMMER 

■CHARACTERISTICS 


SOURCE LANGUAGE 
SOFTWARE CHARACTERISTICS 

Instruction Set 
Self-Documentation 



CHARACTERISTICS 

Input Edits 
Report Format 


Figure 1—The computer programming environment 


4. Total number of edits to be performed on input data 

5. Total number of output files and report formats 

6. Total number of types of output records 

7. Total number of unique fields of output records 

8. Total number of output fields not appearing as fields 
on input records 

9. Total number of input files for which presence is op¬ 
tional 

10. Total number of input records for which presence is 
optional 

11. Total number of output files for which development 
is optional 

12. Total number of output records for which develop¬ 
ment is optional 

13. Total number of control breaks and/or totals specified 
for all output reports 

14. Optional control breaks and/or totals 

15. Mathematical operations other than developing totals 

The programmer characteristics considered as possibly 
having an impact on program development time were simi¬ 
larly believed to be those with the greatest likelihood of 
having an impact on final program size. Those programmer 
variables are: 

1. Number of months of programming experience 

2. Number of months of programming experience using 
the COBOL language 


3. Number of months of experience using the specific 
COBOL language compiler to be used for the subject 
program 

4. Number of months experience programming business 
applications 

5. Number of months experience programming for the 
current employer 

METHODOLOGY 

The purpose of this research was to examine the relation¬ 
ship between various processing characteristics of programs 
and experience characteristics of programmers and program 
size. The ultimate objective was to develop a viable tech¬ 
nique for predicting the amount of source language lines and 
object language bytes a program would eventually become. 

Research hypotheses 

Since there is little research which verified that the pro¬ 
gram and programmer characteristics discussed have an ef¬ 
fect on program size, this research is actually testing the 
null hypothesis, for each specified program variable, that 
the variable has no effect on program size versus the alter¬ 
native hypothesis that the variable is associated with an 
increase in program size, that is, there is a positive corre¬ 
lation between each program variable and program size. 
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The specified programmer variables, it is hypothesized, 
will also be found to be related to program size, but the 
coefficients will have a negative sign, that is, there will be 
a negative correlation. Increasing values for programmer 
variables will be associated with reductions in program size. 


Data collection techniques 

The first step was to locate a data processing facility 
which employed a group of computer personnel to create 
new COBOL programs in an offline environment. The sub¬ 
ject firm was selected for two additional reasons. First, there 
was a wide range of experience among the members of the 
programming staff, from trainees with eight months experi¬ 
ence to senior programmers with over ten years of experi¬ 
ence. Second, the firm also performed contract program¬ 
ming for other firms. This implied that one could expect 
more variation among the types of programs developed at 
this facility than in an installation where programs were 
created only for internal systems. 

It should be restated here that the program variables to 
be measured must, if a predictive technique is to be reliable 
and effective, be measurable before programming begins. 
Since the quality of pre-programming documentation at the 
subject firm was not adequate to allow such measurements 
to be made, it was necessary for the researcher to examine 
the COBOL source statement or compilation listing for each 
program and measure the variables of interest from the list¬ 
ing. While this was necessarily tedious, it also improved the 
control over the research data. This improvement in control 
was accomplished because one could verify that the program 
was indeed coded in COBOL. Also, one was able to detect 
and remove from the analysis any program which utilized 
the Report Writer feature and was therefore not comparable 
to the other programs. In order to measure the value of the 
programmer variables, a questionnaire which asked the pro¬ 
grammers to report their programming experience was cir¬ 
culated to all programmers by the programming manager. 

The size of the program in source language lines of code 
was measured from the listings, subdivided into Data Divi¬ 
sion lines and Procedure Division lines. The number of bytes 
of storage required to store the object code was also taken 
from the compilation listing and was similarly subdivided 
into Working Storage bytes and Procedure Division bytes. 


Statistical analyses to be performed 

The first step would be to provide the data for program 
size and program variables to a stepwise multiple linear 
regression computer program for analysis of the interrela¬ 
tionships present. The identical procedure would be fol¬ 
lowed relating the size of a program to each of the program¬ 
mer variables for the programmer who created the program. 
With regard to the programmer variables, however, it would 
be necessary to adjust the values depending upon the date 


the programmer started the programming assignment. That 
is, the value of each programmer experience variable had to 
be reduced to what the value would have been when the 
programmer started the task since the variables are all 
length-of-time experience values. 

The last step would be to relate program size to the group 
of program variables and programmer variables simultane¬ 
ously to develop a specific predictive equation for use as an 
estimating device. 


Limitations of the study 

The major limitation of the study is the lack of enforce¬ 
ment of programming standards at the firm used in the study. 
Each programmer had his own style in terms of the appear¬ 
ance of the source language code. For example, some pro¬ 
grammers would place both the PICTURE and VALUE 
clauses for a data field on one card or line of code while 
others would place the VALUE clause entry on a separate 
card. Another observable difference was that some program¬ 
mers would place only one verb per line of code in the 
Procedure Division while others would place as much code 
as possible on each card. Another limitation is the accuracy 
with which programmers reported their experience on their 
questionnaire. Aside from the fact that inaccuracies may be 
intentionally introduced into the data, there is the problem 
that a subject was asked to recall how long he performed a 
given type of programming. Further, if a subject performed 
some functions other than those of a programmer during a 
given time period, he was also asked to parcel out that 
portion associated with only the programming effort. Since 
future applications of this technique will be using input with 
these same shortcomings, however, perhaps the reliability 
of the technique will be improved by the high similarity 
between the experimental environment and the application 
environment. 

Some assumptions which could be classified as limitations 
are that the subject programmers were provided with com¬ 
parable documentation of the processing requirements for 
each program and that the programs developed were of 
comparable quality. 

It should be noted that the items listed as limitations are 
sources of the error or unexplained portion of the variance 
to be found in the analysis. Therefore, the mathematical 
statements of the relationships found between the sample 
variables could be considered to be underestimates of the 
true strength of the relationships between the population 
variables. 


RESEARCH FINDINGS 

A total of thirty-six programs were found to qualify for 
statistical analysis by being written in COBOL and contain¬ 
ing no use of special compiler features which made them 
unusual in terms of coding required or logical construction. 
The results of the analysis are stated below. 
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TABLE I—Correlation Coefficients of Program Characteristics with Data 
Division Source Statements 


Program Characteristic r 


Input Files 

.19910 

Input Records 

43704**** 

Input Fields 

57080***** 

Input Edits 

.19552 

Output Files and Report Formats 

47542 **** 

Output Records 

.62756***** 

Output Fields 

72589***** 

Output Fields without corresponding input fields 

.40557*** 

Optional Input Records 

.31012* 

Optional Output Files and Report Formats 

-.15245 

Optional Output Records 

.19226 

Control Breaks and Totals 

.50122**** 

Optional Control Breaks and Totals 

-.08216 

Mathematical Operations 

.37451** 

**** ♦significant at .0005 level 


♦♦♦♦significant at .005 level 


***significant at .01 level 


** significant at .025 level 


♦significant at .05 level 



Program characteristics 


As can be seen in Table I, the most important program 
variable in terms of its impact on Data Division source 
language lines of code was output fields, closely followed 
by output records, input fields and control breaks and totals. 
All of these variables were significant at the .005 level. As 
can be seen, most of the program variables were related to 
an increase in the size of the source code found in the Data 

Division. 


As shown by Table II, the three program variables which 
one could reasonably associate with computations and log¬ 
ical complexity emerged as having the greatest effect on the 
amount of source language code found in the Procedure 
Division. All three, mathematical operations, output fields 
without corresponding input fields and control breaks and 

totals exhibited correlation coefficients of over 

.9. 

TABLE II—Correlation Coefficients of Program Characteristics with 

Procedure Division Source Statements 


Program Characteristic 

r 

Input Files 

.02436 

Input Records 

.09295 

Input Fields 

.01439 

Input Edits 

.02551 

Output Files and Report Formats 

-.03108 

Output Records 

.10602 

Output Fields 

.08491 

Output Fields without corresponding input fields 

.95237* 

Optional Input Records 

.04856 

Optional Output Files and Report Formats 

-.07483 

Optional Output Records 

.03558 

Control Breaks and Totals 

.93371* 

Optional Control Breaks and Totals 

-.04447 

Mathematical Operations 

.97183* 


TABLE III—Correlation Coefficient of Each Program Characteristic with 
Working Storage Bytes 


Program Characteristic 

r 

Input Files 

.58296*** 

Input Records 

.10407 

Input Fields 

-.03186 

Input Edits 

-.03899 

Output Files and Report Formats 

-.00679 

Output Records 

.07999 

Output Fields 

-.00548 

Output Fields without corresponding input fields 

-.04902 

Optional Input Records 

-.05390 

Optional Output Files 

.28230* 

Optional Output Records 

.15972 

Control Breaks and Totals 

-.07603 

Optional Control Breaks and Totals 

.34933** 

Mathematical Operations 

-.05159 

***significant at .0005 level 


**significant at .025 level 
♦significant at .05 level 


In terms of the impact of the program characteristics on 

the amount of object code generated. 

Tables III and IV 

provide considerable insight. The only variables found to be 
strongly related to an increase in the amount of Working 

Storage bytes needed for a program, 

in order of their 

strength, were input files, optional control breaks and totals 
and optional output files. With regard to the Procedure Di¬ 
vision bytes required, the same variables, mathematical op¬ 
erations, output fields without corresponding input fields 
and control breaks and totals are the only program charac- 

teristics which appear to have an effect. 


Programmer characteristics 


Although thirty-six programs were available for analysis 
regarding the impact of program characteristics on program¬ 
ming time, five of those had to be removed from the analysis 
of programmer variables because the responsible program¬ 
mers had left the firm and their characteristics were not 


TABLE IV—Correlation Coefficient of Each Program Characteristic with 
Procedure Division Bytes 

Program Characteristic 

r 

Input Files 

.02861 

Input Records 

.04875 

Input Fields 

-.02495 

Input Edits 

.01090 

Output Files and Report Formats 

-.05898 

Output Records 

.07655 

Output Fields 

.04933 

Output Fields without corresponding input fields 

.96540* 

Optional Input Records 

.00096 

Optional Output Files 

.06858 

Optional Output Records 

.01659 

Control Breaks and Totals 

.94325* 

Optional Control Breaks and Totals 

-.03942 

Mathematical Operations 

.98177* 


significant at .0005 level 


‘significant at .0005 level 
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TABLE V—Correlation Coefficients of Programmer Characteristics with 
Data Division Source Statements 


Programmer Characteristic r 


Months of experience programming at this facility -.46063** 

Months of experience programming business-oriented -.44684** 
applications 

Months of experience programming (total) -.45020** 

Months of experience programming with COBOL — .27983 

compilers 

Months of experience programming with COBOL -.34568* 

compiler in use at this facility 


**significant at .01 level 
*significant at .05 level 

measurable. For the thirty-one programs, the four measure¬ 
ments of program size were individually regressed against 
the programmer characteristics and the correlation coeffi¬ 
cient between each characteristic and each component of 
program size was computed. 

The findings shown in Table V indicate four of the five 
programmer experience variables are related to reductions 
in the size of the Data Division source code. The three 
strongest influences appear to be total programming expe¬ 
rience, business programming experience and, curiously, 
experience at this facility. While one can easily accept the 
business programming and total programming experience 
variables as important, the relationship between experience 
at this facility and reduced Data Division code is less clear 
and may be unique to this data processing installation for 
reasons unknown at this time. 

The analysis of programmer characteristics and source 
statements found in the Procedure Division provided the 
most unexpected results. As the correlation coefficients in 
Table VI indicate, there appears to be no significant rela¬ 
tionship between any programmer experience characteristic 
and the size of the source language Procedure Division. 
Since previous research 4 has found these programmer vari¬ 
ables are all significantly related to reduced program devel¬ 
opment time, this latter finding raises an interesting ques¬ 
tion. Does increased experience of the types listed give one 
the ability to develop the same amount of processing logic 
code as another less-experienced programmer in signifi¬ 
cantly less time? That is the implication of these findings. 

When relating the programmer characteristics to the ob¬ 
ject code equivalents of the Data and Procedure Divisions 
source code, the findings were as shown in Tables VII and 

TABLE VI—Correlation Coefficients of Programmer Characteristics with 
Procedure Division Source Statements 


Programmer Characteristic r 

Months of experience programming at this facility .04937 

Months of experience programming business-oriented .00333 

applications 

Months of experience programming (total) -.00152 

Months of experience programming with COBOL .08817 

compilers 

Months of experience programming with COBOL .11644 

compiler in use at this facility 


TABLE VII—Correlation Coefficients of Programmer Characteristics with 
Working Storage Bytes 


Programmer Characteristic r 


Months of experience programming at this facility -.34113* 

Months of experience programming business-oriented -.35143* 
applications 

Months of experience programming (total) -.36714** 

Months of experience programming with COBOL —.30781* 

compilers 

Months of experience programming with COBOL -.28129 

compiler in use at this facility 


**significant at .025 level 
*significant at .05 level 

VIII. The findings with regard to Working Storage bytes 
differ only slightly with the source code findings. Total pro¬ 
gramming experience, business programming experience 
and programming experience at this facility, however, re¬ 
main the most important variables. With regard to Procedure 
Division bytes, the findings are identical to those for the 
impact of the variables on source code, that is, no significant 
relationships were found. 

Estimating equations 

In order to develop equations for estimating the two com¬ 
ponents of source program size and the two components of 
object program size, one would use a stepwise multiple 
linear regression analysis program. The logic of such a pro¬ 
gram is to continue to add each of the predictor variables 
into the estimating equation one by one, each time entering 
that remaining variable which most improves the accuracy 
of the estimate. A point is eventually reached where the 
additional value of the next variable in the equation is small 
and, more importantly, not of statistically significant value. 
The equations shown here are those which in each case 
were, statistically, optimum. In order to develop such equa¬ 
tions, the variables to be subjected to analysis would be 
those which appeared to have a significant impact. 

The approach decided upon was to first use only the group 
of program characteristics to develop equations for the 
source and object code of the Data and Procedure Divisions. 
The analyses would then be repeated allowing programmer 
characteristics to enter the equations to determine if predic¬ 
tive power was increased. The equation shown in Figure 2 
for estimating Data Division source lines had a multiple 


TABLE VIII—Correlation Coefficients of Programmer Characteristics with 
Procedure Division Bytes 


Programmer Characteristic r 


Months of experience programming at this facility .07447 

Months of experience programming business-oriented .03104 

applications 

Months of experience programming (total) .02666 

Months of experience programming with COBOL compilers .11312 

Months of experience programming with COBOL compiler . 14020 

in use at this facility 
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Variable Name 

Regression 

Coefficient 

Constant 

-26.41 

Output Fields 

1.69 

Output Fields without corresponding input fields 

4.84 

Input Files 

35.45 

Input Fields 

.59 

Mathematical Operations 

-.62 

Input Records 

6.03 


Figure 2—Equation for predicting data division source lines with program 
characteristics 


correlation coefficient of .94 and, therefore, the variables 
included explain about 88 percent of the variance found in 
the number of source lines appearing in the Data Division 
of the sample programs. The optimum equation for the Pro¬ 
cedure Division source lines, shown in Figure 3, using only 
program characteristics, had a multiple correlation coeffi¬ 
cient of .997. 

As an example of the use of these equations, assume one 
wished to estimate how many source lines of code would be 
required for a program with the following characteristics: 

8 mathematical operations 
20 output fields 
1 input record type 
0 optional output records 
0 input edits 

4 output fields without corresponding input fields 
18 input fields 
and 

1 input file 

By placing those values in their appropriate positions in the 
equations one obtains a value of about 74 lines of Data 
Division code and 140 lines of Procedure Division code. The 
program would then be estimated at about 214 lines of code 
plus the necessary Identification and Environment Division 
entries, which typically show little variance. Upon allowing 
the programmer characteristics to enter the equations it was 
found that, although the equations became more complex, 
no significant improvement could be made in the estimate. 

The next analysis concerned the object code components 
of the program. The program variables which developed the 


Variable Name 

Regression 

Coefficient 

Constant 

-31.09 

Mathematical Operations 

2.97 

Output Fields 

1.81 

Input Records 

37.02 

Optional Output Records 

45.98 

Input Edits 

-2.32 

Output Fields without corresponding input fields 

2.90 


Figure 3—Equation for predicting procedure division source lines with 
program characteristics 


Regression 

Variable Name Coefficient 


Constant -6019.57 

Input Files 7377.95 

Optional Control Breaks and Totals 1802.77 

Output Files and Report Formats -1903.16 


Figure 4—Equation for predicting working storage bytes with program 
characteristics 


best equation, statistically speaking, are shown in Figures 
4 and 5. The multiple correlation coefficient for the Working 
Storage bytes equation was .66, accounting for only about 
43 percent of the variance in this component of program 
size. Once again it was found that allowing programmer 
characteristics to enter the equation provided little addi¬ 
tional predictive power. 

The optimum equation for predicting Procedure Division 
bytes using only program characteristics, shown in Figure 
5, had a multiple correlation coefficient of .998. This implies 
that the number of bytes required in the Procedure Division 
is apparently predictable from knowledge of the program 
characteristics appearing in Figure 5. Programmer charac¬ 
teristics were once more found of no significant value to the 
equation. A necessary but realistic assumption is that the 
programming specifications at a data processing facility are 
of sufficient quality that the program characteristics in the 
equations are defined before programming is begun. Given 
this assumption is met, the equations presented in Figures 
4 and 5 could be used to predict the amount of storage a 
program will require by placing the program characteristics 
in the equations and computing the final value as shown 
previously for the source statement portions of a program. 

SUMMARY AND CONCLUSIONS 
Summary 

The purpose of this research was to analyze the relationships 
between characteristics of a programming problem and the 
responsible programmer and the amount of source and ob¬ 
ject code necessary to complete the programming problem. 
For each program characteristic it was hypothesized that 


Variable Name 

Regression 

Coefficient 

Constant 

-38.64 

Mathematical Operations 

50.37 

Output Fields 

26.32 

Optional Output Records 

626.73 

Input Records 

659.70 

Output Fields without corresponding input fields 

63.57 

Input Edits 

-19.68 

Optional Input Records 

-218.67 


Figure 5—Equation for predicting procedure division bytes with program 
characteristics 
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an increase in the value of that variable would lead to an 
increase in each aspect of program size. For each program¬ 
mer variable it was believed that an increase in the value of 
the variable would contribute to a decrease in the program 
size measurements. 

Simple linear correlation coefficients were computed to 
determine the direction and magnitude of the relationships 
between the variables in the analysis. In order to develop 
predictive equations for the components of program size, 
stepwise multiple linear regression analyses were per¬ 
formed. 

Conclusions 

Of the fifteen program characteristics examined, nine were 
found to have correlation coefficients with Data Division 
source statements significant beyond the .05 level, while 
only three were found correlated with Procedure Division 
source statements. The strength of the latter relationships, 
however, was significant beyond the .0005 level. 

In terms of their impact on Working Storage bytes of 
object code, only three program characteristics were found 
to be significant, varying in strength from only .05 to .0005 
levels of significance. Three different program characteris¬ 
tics, however, were found to be related to Procedure Divi¬ 
sion bytes of object code, all at the .0005 level of signifi¬ 
cance. 

While all five programmer characteristics were negatively 
correlated with both Data Division source statements and 
Working Storage bytes as hypothesized, only four were sig¬ 
nificant beyond the .05 level. None of the five programmer 
experience characteristics could be shown to cause a reduc¬ 
tion in either the Procedure Division source lines of code or 
the Procedure Division bytes of object code, a finding wor¬ 
thy of note. 

When the program and programmer variables were used 
to develop predictive equations for the four size compo¬ 
nents, strong multiple correlation coefficients were found 
without programmer characteristics present in the equa¬ 
tions. The inclusion of programmer characteristics into the 
equations appeared to provide no significant increase in 
predictive power. 

The findings of this study imply that one could, with a 
reasonable degree of accuracy, predict the two major source 
and object components of program size of a program before 
programming begins by developing equations of the type 
presented here with data collected at the facility where they 
are to be used for estimation. As new programs are devel¬ 
oped, their program and programmer characteristics and 
four size components should be measured and added to a 
history data bank for the firm. These data should then be 


periodically used as input to a stepwise multiple linear 
regression analysis program. As the firm continues to in¬ 
crease the number of program data points in its data bank, 
it will be capable of predicting the size of new programs 
with increasing accuracy. 
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Multiprocessing made easy 

by RONALD J. PRICE 

Perkin-Elmer Data Systems Group 
Tinton Falls, New Jersey 


INTRODUCTION 

The concurrent execution of processes on multiple machines 
has appealed to system architects for quite some time.* In 
fact, it is not such a new concept as often thought. According 
to P. Enslow, “The first operational ‘true’ multiprocessor 
was delivered in 1962.” 1 

Increased throughput is usually the major objective of 
multiprocessing, but there are other advantages over inter¬ 
leaved, multi-programming of a uniprocessor that oftentimes 
take on greater importance. 

For example: 

• Resource sharing 

• Responsiveness to external events 

• Expandability in a stepwise manner 

• System modularity 

• Survivability 

There are a number of reasons why multiprocessing is not 
as popular as perhaps it could be, as Enslow aptly points 
out in Reference 2. One big problem is software. Program 
development is difficult because the behavior of concurrent 
programs is time-dependent and seldom reproducible with 
a given set of input data. Add parallelism with increased 
possibilities for time-dependent interactions and deadlock 
situations, and we have a most complex problem. The lack 
of software for the number of available multiprocessor sys¬ 
tems 1 attests to this fact. 

Help is on the scene, though! 

A new design concept that combines shared data struc¬ 
tures and the operations permitted on them in a single syn¬ 
tactical construct called a “monitor” has been suggested in 
the literature by C. A. R. Hoare 3 and Per Brinch Hansen 4 
as an effective tool for building concurrent system programs. 
Brinch Hansen based his Concurrent Pascal language on this 
principle. He extended the sequential program properties of 
the Pascal language with concurrent programming tools 
called processes and monitors. 5 This paper is based on the 


* Tightly coupled, multi-port shared memory systems are assumed in this 
paper, although much of the presentation equally applies to other types of 
multi-processor systems. 


constructs in Concurrent Pascal, although the research 
works of E. W. Dijkstra 6 and N. Wirth 7 are also relevant. 

Evidence exists that the monitor concept, particularly as 
implemented in Concurrent Pascal, is a powerful tool. 
Brinch Hansen recorded improvement in programmer pro¬ 
ductivity while building a complete operating system with 
his language. 8 The utility of the language has also been tested 
for many diverse applications. 9 

From personal experience, the author has found that the 
mere act of decomposing a concurrent problem into logical 
processes and monitor modules, irrespective of how they 
might be implemented, is an excellent design methodology. 
Furthermore, once system facilities are provided to manage 
process and monitor programs, the systems programmer 
need no longer suffer through analyses of time-dependent 
race conditions and interprocess communication protocols. 
Performance need not be sacrificed either. In fact, the author 
discovered that performance improved, probably because of 
a better overall system design. 

The monitor concept is a relatively new idea and some 
weaknesses can be identified. Several researchers seek ad¬ 
ditional language constructs and operators (for examples see 
References 9 through 14). This is to say nothing of the age- 
old question whether compilers can produce optimum time 
and space code, nor the difficulties of implementing language 
support for multiprocessing. On the other hand, the system 
programmer need not wait for the ultimate language. He can 
reap the benefits of this new software technology for current 
problems even if he has to use assembly language. All he 
has to do is build a kernel facility, which manages process 
and monitor programs, and adhere to the programming con¬ 
ventions promulgated by these experimental languages. 

The next section presents a brief explanation of the mon¬ 
itor concept and how it relates to multiprocessing applica¬ 
tions. This is followed by a section on implementation tech¬ 
niques for supporting multiprocessors. The last section 
illustrates an example application and its solution. By em¬ 
ploying two assembly language solutions, one with monitors 
and one without, the example evaluates the fundamental 
utility of the monitor concept for solving concurrent/multi- 
processing problems; i.e., biases of a compiler for generating 
code and for enforcing programming conventions are not 
involved. The example also illustrates the value of the mon¬ 
itor as a systems structured programming tool by solving 
the concurrent problem in stages. 
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MONITOR CONCEPT 

The reader should consult the references for an in-depth 
explanation of monitors and how they are employed in con¬ 
current programs. Briefly though, monitors consist of shared 
data structures combined with a well-defined set of proce¬ 
dures permitted to operate on the shared data. In this con¬ 
text, processes are sequential programs which synchronize 
their operations and get access to shared data structures by 
calling monitor procedures in a manner much like subrou¬ 
tines. A run-time kernel executive enforces mutual exclusion 
on access to a monitor from competing processes. 

The simple example in Figure 1 illustrates how processes 
can asynchronously exchange messages through monitors. 
Input and output operations are overlapped concurrently 
with job processing. The INPUT process READs a message 
from the IN_TERMINAL and PUSHes it on a buffer stack 
for the JOB process in the IN_BUFF monitor; a similar 
operation takes place on output. The monitors can be rep¬ 
resented by ordinary program code except they are passive 
modules and do not execute until called. They contain the 
procedures (for example, PUSH and GET) that operate on 
the shared buffer stacks. They also contain the counters and 
flags needed to control the operation. 

Inter-process synchronization is achieved through the use 
of a delay-queue construct by which a monitor procedure 
can delay its calling process or continue another process 
waiting in a delay-queue. For example, in Figure 1 the JOB 
process knows nothing of the situation of the INPUT and 
OUTPUT processes. If it tries to GET a message when none 
is available, it is delayed by IN_BUFF until continued when 
one is supplied by the INPUT process. Likewise, if the JOB 
process tries to PUT an output message when no buffer is 
available for the data, it is delayed by OUT_BUFF until the 
OUTPUT process FETCHes a previous message and frees 
a buffer. 

Other applications for monitors include classical reader- 
writer problems, process synchronization on events, re¬ 
source sharing, priority scheduling, and so on. The concept 
is valid because processes, containing their own private data 
structures, cannot directly intercommunicate, change con¬ 
trol (shared) data, or access a shared resource; to do so they 
must call a monitor, and the kernel permits a monitor to 
service only one process at a time. 

The technique of describing a system program in terms of 
monitors and sequential processes can be used to represent 
a multiprocessing program. Any process is a candidate to 
be executed in parallel with others on separate processors. 
The three processes in Figure 1 could be executed literally 
simultaneously on different processors with no more diffi¬ 
culty (functionally) than if interleaved concurrently on one 
processor. Their combined execution becomes serial only 
when interacting through the same monitor. 

In a multiprocessor configuration, monitor code can be 
executed out of shared memory or be replicated in each 
processor’s private local memory. That is, if necessary, an 
instance of a given monitor can be represented on multiple 
machines. The separate representations can even be imple- 



IN-TERMINAL OUT-TERMINAL 

Figure 1—Example application of monitors 


mented differently (for example, microcoded) in different 
processors provided they execute the procedures as required 
by their respective processes. Monitor code can be dupli¬ 
cated in different processors because the run-time kernel 
permits only one version to execute at any given time. How¬ 
ever, only one copy of a monitor’s variable data structures 
can exist and it must be located in shared memory to be 
accessible by the different processors. 

Each physical processor contains a run-time kernel. Col¬ 
lectively, these kernels create a virtual machine which sup¬ 
ports an integral distributed processing system. In fact, con¬ 
trol over processes for scheduling and dispatching is shared 
among the processors and consequently the virtual machine 
can be considered truly distributed. 15 

For example, the collection of the processes and monitors 
in Figure 1 represents a concurrent system program that 
accomplishes an application function (albeit rather trivial in 
this case). It consists of disjoint modules which run in har¬ 
mony as one program where parts of it may execute in 
parallel. No single processor controls the execution of the 
others. With adequate kernel support, the application pro¬ 
gram can be executed on multiple machines or on a single 
machine without change to any of its code. 

In short, the use of monitors in this manner reduces the 
software implementation effort for multiprocessor systems 
to the scope of single-processor designs. Or from another 
viewpoint, greater degrees of performance can be achieved 
from one basic design simply by adding processors. 
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IMPLEMENTATION NOTES 

A run-time kernel is needed to manage concurrent pro¬ 
grams according to the principles of Concurrent Pascal. The 
kernel is responsible for waiting and dispatching processes 
(sometimes called short-term scheduling), and essentially is 
a miniature executive. In this case, it also recognizes mon¬ 
itor program modules and provides exclusive access to their 
respective shared data structures. Additionally, the kernel 
manages delay-queues and implements the delay and con¬ 
tinue operators. 

Given the kernel, the systems programmer builds an ap¬ 
plication program by coding appropriate process and moni¬ 
tor modules that solve his problem. These program modules 
are then linked together and loaded into the computer with 
the kernel. These steps can be automated with a compiler. 
In fact, the existence of the kernel is transparent to a pro¬ 
gram written in Concurrent Pascal or equivalent language. 
(Concurrent Pascal also provides strong type checking and 
a notion of access rights, both of which are outside the 
scope of this paper.) 

Space does not permit documenting a complete kernel, 
but Figure 2 is a basic flowchart of four key kernel routines. 
Illustrated are mechanisms for entering and exiting a monitor 
and for implementing the delay and continue operators. 
These routines are executed in behalf of the using program 
when requesting the services of the kernel. The DELAY 
and CONTINUE routines in Figure 2 adhere to the conven¬ 
tions of Concurrent Pascal; note that CONTINUE implies 
EXIT as well. Alternative kernel designs are possible, as 
deemed appropriate for given installation objectives and re¬ 
quirements. 

Support for multiprocessors is reflected in Figure 2 in 
several ways. In particular, the kernels (one in each ma¬ 
chine) share data structures located in shared memory to 
control their operations. They need a mechanism for resolv¬ 
ing contention when more than one kernel tries to access a 
common kernel data structure. This can happen when se¬ 
lecting a process to dispatch and when attempting to grant 
access to a monitor for a process. A hardware read-modify- 
write function (Test and Set instruction) is employed to lock 
a data structure before a kernel accesses it. When a kernel 
finds a structure busy, it “spins” on the lock, testing it until 
it becomes free. Locking is depicted in Figure 2 by boxes 
with titles. The kernel need not necessarily treat a busy lock 
as a busy wait situation, but it is not likely to have anything 
else to do except spin. 

Processor time consumed while spinning on a lock is data 
contention overhead and will degrade total system perform¬ 
ance if the frequency of accessing a given structure is high 
among the processors. Probability of contention can be min¬ 
imized by dedicating a separate lock for each monitor and 
for each process dispatch chain in the system. Although the 
lock intervals can become a bottleneck when average proc¬ 
ess execution time between kernel calls approaches the ex¬ 
ecution time of the kernel routines, this is generally not the 
case because the lock intervals are relatively short. For 
example, in Figure 2 the lock interval in the ENTER MON¬ 


ITOR routine is not for the duration of the monitor proce¬ 
dure, but rather only for the short interval the kernel takes 
to interrogate the monitor’s busy flag and to chain the proc¬ 
ess to the monitor if the monitor is busy with another proc¬ 
ess. Upon exiting a process from a monitor, the kernel 
interrogates the monitor’s busy chain and makes a waiting 
process (if any) ready for executing the monitor. 

A process can be either dedicated or shared. The code for 
a shared process is located in shared memory and can be 
executed by any processor in the system; a dedicated proc¬ 
ess is located in a processor’s private local memory. A 
shared process has an advantage over a dedicated process 
in that it can be dispatched by any processor in the system, 
whereas a dedicated process might be in a ready state but 
unavailable to idle processors in the system. While there are 
a number of reasons for having shared processes, they might 
execute slower than dedicated processes because of memory 
access contention and particular hardware constraints of a 
given installation. Processes might also have to be dedicated 
if they frequently access resources (peripherals, clocks, sen¬ 
sor equipment, etc.) that are accessible only on a given 
processor. 

Process dispatching is distributed (except for the control 
structures in shared memory) because any processor’s ker¬ 
nel can set processes ready for execution on other proces¬ 
sors in the system. As illustrated in Figure 2, this can happen 
when a process waiting on a monitor busy chain is activated 
and when a process waiting in a delay-queue is continued. 
No single kernel is responsible, nor need be interrupted, for 
deciding the order of putting processes in a ready state. 
Wdien a kernel selects a process for dispatching it selects 
the highest priority process that it is capable of executing 
(processes inside a monitor have priority over processes not 
in monitor mode). 

Figure 2 reflects some design choices that would not be 
required for a single processor system. For example, when 
continuing a process waiting in a delay-queue, the kernel 
might not be able to switch immediately to the continued 
process because it might be dedicated to another processor. 
So the kernel routine falls through Dispatch instead. In fact, 
Dispatch is exercised whenever the kernel has the oppor¬ 
tunity in order to insure fair priority treatment of processes 
in monitor mode. Otherwise, ready processes inside moni¬ 
tors might be blocking resources needed by other processes 6 
and therefore idling processors. 

An inter-processor interrupt facility enables a kernel in 
one processor to signal the kernel in another processor when 
setting one of its processes ready for execution. All idle 
processors can be alerted when readying a shared process. 
The interrupt need not convey any information except to 
cause the recipient kernel to cycle through its Dispatch 
routine. If processes are not to be preempted, which has 
merits germane with the philosophy of the monitor con¬ 
cept, 16 ’ 17 the interprocessor interrupt need be generated only 
when the destination processor is idle (or when the readied 
process is in monitor mode, if so desired to allow monitors 
to preempt processes). The interrupt facility is not an ab¬ 
solute requirement, however. When in idle mode with no 
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Figure 2—Kernel procedures to support monitors 
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work to perform, the kernel could continually cycle through 
its dispatch chains instead of going into a wait state (or 
perhaps periodically on a clock interval). 


EXAMPLE SYSTEM 

An example problem is presented to illustrate how mon¬ 
itors can be employed in building non-trivial concurrent 
programs. The utility of the monitor concept is evaluated by 
comparing the results with another solution of the same 
problem that did not employ monitors. Performance meas¬ 
urements are also given of various test runs when applying 
the monitor solution, as a multiprocessing program, to a 
dual processor configuration. 

The following example application was defined as a 
benchmark model for investigating kernel designs and for 
evaluating relevant performance aspects. The benchmark is 
intended to reflect much of the character of a highly inter¬ 
active transaction processing system, but actually it is a 
hypothetical application wherein data formats and process¬ 
ing functions are kept to a minimum (simulated) in order not 
to obscure the inherent interactions in the application prob¬ 
lem being solved. 

The benchmark application was implemented twice on an 
existing multi-tasking operating system in assembly lan¬ 
guage. Conventional operating system features for interpro¬ 
cess communications and synchronization were employed 
in the first implementation. This was done before any mon¬ 
itor capability was even designed. The second case em¬ 
ployed monitors. A kernel facility employing the techniques 
described in the previous section was developed to run 
under the environment of an operating system task. 

The application can be briefly described as terminal op¬ 
erators being able to exchange messages and to start con¬ 
current job processes which execute disc I/O and computing 
functions according to the operator’s request. The job pro¬ 
grams send their results to the terminal specified in the 
originator’s message. Job operations and terminal message 
exchanges run concurrently. Terminal I/O is overlapped so 
that an operator can input a message while the previous 
message is being processed. Output messages preempt 
input; job output takes presidence over terminal messages; 
job requests are serviced first-in-first-out. Each message 
delivered to a terminal contains a header identifying the 
source unless the previous output was from the same source. 

Only the monitor approach to the benchmark problem is 
described here. Documenting the solution employing con¬ 
ventional techniques would serve no useful purpose. The 
monitor solution is itself somewhat arbitrary; there are prob¬ 
ably many other ways of solving the benchmark problem in 
terms of monitors and processes. The approach taken here 
attempts to balance performance with conservation of mem¬ 
ory. The benchmark program is built up in stages to simplify 
testing. Five terminals and two job processes are supported. 

The benchmark program is described in pictorial form in 
Figures 3 through 5 with circles representing processes and 
boxes representing monitors. Program modules are identi¬ 
fied by a global name and a type name in parentheses. 


TERM 



Figure 3—Benchmark terminal subsystem 


Program modules of the same type have identical code; their 
data structures are replicated during program integration to 
build separate instances of the same module type. 

The first stage consists of a single terminal subsystem as 
illustrated in Figure 3. The INPUT process continually in¬ 
puts messages and passes them on to the message EX¬ 
CHANGE monitor. The OUTPUT process continually gets 
messages from the EXCHANGE monitor when available 
and outputs them to the terminal. The terminal resource 
CONTROL monitor is responsible for granting exclusive ac¬ 
cess to the terminal and for maintaining its read/write/idle 
mode. The CONTROL monitor preempts a pending read 
I/O by executing halt I/O if the OUTPUT process requests 
access while the terminal is in read mode. The purpose of 
the READ monitor is to insure, by being a higher priority 
program module than the OUTPUT process, that the read 
I/O will be physically posted before the OUTPUT process 
can attempt to preempt it and cause contention problems 
with halt I/O. The CONTROL monitor also writes the 
prompt character while granting read access to avoid poten¬ 
tial contention with halt I/O. 

The next stage is to add the JOB processes as illustrated 
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in Figure 4. The aspects of controlling the JOB processes is 
much the same as for the OUTPUT process. However, with 
the addition of the JOB processes, the CONTROL monitor 
must maintain the identification of the last writer in order to 
indicate whether or not a header message is required on a 
new output request. It must also provide a delay queue for 
the JOB processes separate from the OUTPUT process in 
order to give priority to job output versus terminal output. 
Separate EXCHANGE monitors could have been provided 
for the JOB and OUTPUT processes, but a combined mon¬ 
itor approach was selected to conserve memory. Code in 
the EXCHANGE monitor is kept to a minimum to avoid 
contention losses in multiprocessor configurations. 

Adding terminals to Figure 4 impacts the buffer require¬ 
ments of the EXCHANGE monitor. It must provide an 
exchange mechanism for each source-destination pair, times 
two to allow overlapped I/O. This number in terms of buffers 
is not actually required by the application (in fact two times 
the number of terminals is adequate), so a buffer pool is 
employed in EXCHANGE to dispense message buffers, two 
per INPUT at most. With the addition of Pop and Push 
procedure calls in EXCHANGE for obtaining and releasing 
buffers, the full benchmark program is illustrated in Figure 
5 (for five terminals and two job processes). 

Although this third stage appears to be complex, it is 
actually a trivial extension. All complex logic (of CONTROL 


and EXCHANGE in particular) was verified in the previous 
two stages (CONTROL and EXCHANGE were initially 
written with all the features needed in the final stage). 

The total application consists of: 

• 5 input processes of type IN 

• 5 output processes of type OUT 

• 2 job processes of type JOB 

• 5 read monitors of type RD 

• 5 control monitors of type CTRL 

• 1 exchange monitor of type MGR 

• 5 logical terminals of type TERM 

• 2 logical devices of type DISC 

• 27 delay queues (not described) 

Clearly the system is highly modular and structured. 

Comparisons between the two implementation philoso¬ 
phies for a single processor configuration are summarized 
in Table I. Development effort for the kernel is not included 
in Table I, as it is a generalized facility taking the place of 
the tools otherwise provided by the conventional operating 
system. 

Although the design effort was the same, coding and test¬ 
ing of the monitor solution took less time which resulted in 
nearly a doubling of productivity. Documentation, accept¬ 
ance testing, and other activities normally encountered in a 
production environment are not included in these figures; 
however, the overall improvement should still be significant 
considering the understandability, modularity, and struc¬ 
tured design of the monitor approach. While developing the 
monitor solution to the benchmark application, we experi¬ 
enced no aggravation in resolving race conditions and in 
defining correct inter-process communication protocols as 
occurred in the conventional case. The translation from de¬ 
sign to code was also smoother, and the design itself was 
much more lucid. For example, when running tests to meas¬ 
ure minimum buffering requirements we were unable to ex¬ 
plain the conventional-system results, but the monitor-sys¬ 
tem results were predictable. 


TABLE I—Benchmark Comparisons 



Conventional 

System 

Monitor 

System 

Development Time (work-days) 

Design 

5 

5 

Code 

11 

5 

Test 

9 

3 

Total 

25 

13 

Memory Size (1,000 bytes) 

Sharable code 

3 


Total including data (monitor system 

im 

13 Va (note) 

includes over V 2 K for 
multiprocessing) 

Performance Tests (messages per unit 
time) 

User transactions with minimum job 
processing 

157 

192 

Execute-bound exchanges 

273 

416 

I/O-bound exchanges 

143 

228 
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The two solutions required approximately the same 
amount of memory. It should be noted that the kernel for 
the monitor solution was designed to support multiprocess¬ 
ing while the conventional-system was intended to run only 
on a single processor. The monitor solution therefore in¬ 
curred unnecessary overhead when applied to a single pro¬ 
cessor. 

The single processor monitor-system ran considerably 
faster than the conventional-system. The execute-bound test 
simulated maximum loading for terminal to terminal message 
exchanges. The I/O-bound test was performed with a single 
high-speed “terminal” (magnetic tape) sending to a low 
speed terminal. The better performance of the monitor so¬ 
lution, even with its resident multiprocessing overhead, is 
attributed to a better design and to more efficient inter¬ 
process communications with monitors versus existing op¬ 
erating system features. 

The conventional-system was not designed to support 
multiprocessing because it seemed to be a very laborious 
and unproductive task. The real test of the monitor approach 
came about when we were able to bring up two different 


multiprocessor versions—a dual processor and a four-pro¬ 
cessor configuration with shared JOB processes—of the 
identical application program in only the short time needed 
to build load modules for each machine. After debugging the 
application on a single processor, no errors were uncovered 
during any of the multiprocessor tests on real hardware. 
Indeed, the processes which ran interleaved concurrently 
on a single processor ran in parallel on multiple processors 
without change to any of the application code. 

Performance comparisons between single and dual pro¬ 
cessor configurations are given in Table II. In these tests 
the processors were the same model, and the data structures 
stored in shared memory for the dual processors were lo¬ 
cated in the same physical memory module for the single 
processor tests; furthermore, code was not executed out of 
shared memory. Consequently, memory access times were 
the same for the two configurations. 

In the first test case, messages were exchanged between 
“null” terminals which did not involve real I/O nor much 
processing. The result is that the extra processor helped 
little (33 percent) because the application exhibited little 
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TABLE II—Multiprocessing Comparisons 



Single 

Processor 

Dual Processor 


(Messages per unit time) 

Terminal-to-terminal exchanges with “null” 
devices 

User transactions with job processing (per 

383 

510 

processor) 

182 

181 

Concurrent execution of job transactions 

182 

357 


parallelism in this test run and because data contention 
(spinning on locked data structures) was a bottleneck due to 
the very short execution periods of the application routines. 

The second test represents a more realistic situation. The 
rate of processing user transactions was measured on the 
single processor. In the dual processor case, one processor 
was kept busy executing job computations while the second 
processor processed the same user transactions as in the 
single processor case. The dual processor was able to proc¬ 
ess essentially the same number of messages as the single 
processor (99 percent) along with heavy computing loads. 

The third test was derived from a different, but similar, 
application. It allowed the compute functions of job proc¬ 
essing to be executed in parallel. The dual processor was 
able to process 1.96 times as many messages per unit time 
as the single processor. 

CONCLUSIONS 

We conclude that the monitor concept is indeed an effective 
tool for building concurrent system programs. Furthermore, 
the concept can be applied as a useful design methodology, 
irrespective of the implementation mechanisms. Highly 
modular and structured systems will result with minimal 
opportunities for obscure time-dependent bugs and deadlock 
situations. 

We applied the monitor concept to multiprocessing and 
demonstrated its particular utility for such applications. Sys¬ 
tem programs intended to run on multiprocessor configura¬ 
tions can be reduced to the scope of single processor designs 
by employing monitors. We also illustrated techniques for 
implementing a run-time kernel to support multiprocessing. 
With such a kernel, integrated distributed processing pro¬ 
grams can be built that accommodate different multiple pro¬ 
cessor configurations without change to the application 
code. 
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INTRODUCTION 

C. A. R. Hoare, N. Wirth, E. W. Dijkstra, and some others 
active in structured programing have described and tacitly 
recommended a hierarchical accessibility of passed data in 
structured programing (see for example, References 11 and 
26). This fits well, they and others indicate, with both the 
theory and the practice of structured programing (see for 
example, References 15, 18, 19, and 25). 

This paper does not question that the hierarchical acces¬ 
sibility with parameter passing can be made to work. It may 
even be the most elegant data accessibility for some inter¬ 
esting academic problems. However, this paper does argue 
that for most practical software design situations, other ap¬ 
proaches are on balance superior, especially for the kinds 
of situations commonly implemented using COBOL, FOR¬ 
TRAN, PL/1, JOVIAL, or BASIC. To support this argu¬ 
ment, this paper presents first some definitions of terms, 
and then an examination of some evidence in nine cate¬ 
gories. Following a ranking evaluation, the paper draws a 
conclusion. 


DEFINITIONS 

For purposes here, a system or program designed and 
implemented with structured programing techniques is char¬ 
acterized as having a proper nesting of functions in three 
basic control patterns with single entrance and exit, in the 
sense defined by Mills. 20 This paper does not use “struc¬ 
tured programing” to mean “structured coding,” but rather 
uses it in the sense sketched by Dijkstra. 5 Further, struc¬ 
tured programing is characterized by a hierarchical structure 
of modules (segments) again in the sense defined by Mills, 
which to people appear to specify the performance of spe¬ 
cific single functions, as defined by Pamas and McGowan. 19,22 
The function parsing techniques that lead to the creation of 
such proper-nested structures have been described else¬ 
where. 9 The object is ease of human understanding. 7 * 16 

For the purposes of this paper, a level of abstraction as 
it applies to data is defined to be a grouping of data based 
upon the recognition by people of some common character¬ 
istics, often semantic rather than structural. 11,19 * 24 Data con¬ 
forming to levels of abstraction is referred to in this paper 


as “leveled data.” For example, a file (composed of records 
in turn composed of fields) is regarded as more abstract than 
the component fields individually. In contrast, a disjoint 
string is defined to be a string data structure which has as 
member elements items of data with no necessity of simi¬ 
larity in level of abstraction. Thus a disjoint string might be 
composed of two Boolean variables, a record from a file, 
three tables, and a queue. Pointers may be among the ele¬ 
ments of a disjoint string. 17 Because of the potential diver¬ 
sity, each member element in the string is specifically enu¬ 
merated and described. 

For the purposes of this paper, a flow graph is defined to 
be a graphic presentation of part or all of a control flow in 
a system or program. 1 The edges in the graph represent an 
explicit flow of control unaccompanied by the manipulation 
of data but accompanied by the implicit or explicit com¬ 
munication of data. The nodes in the graph represent an 
implicit flow of control accompanied by successive steps in 
the explicit manipulation of data, and accompanied by some 
means for the communication of data among the steps. 

EXAMINATION 
Access volume potential 

The access volume potential is a count of the maximum 
possible number of data items with assigned values which 
are accessible to any given module upon invocation, 
summed for all modules. Ideally, this count should be small, 
since the larger it is, the greater is the difficulty of human 
understanding in design, implementation, and maintenance. 

The calculation of the access to data items has been de¬ 
scribed by Allen and by Cocke. 1,2 Following “procedure A” 
as an “algorithm for finding intervals,” a partition can be 
made of a control flow graph into basic intervals. If the 
hierarchy of modules is a true tree conforming to a proper 
nesting of functions, then it is possible to reduce selectively 
the basic control graph to yield nested intervals of higher 
and higher order, as diagramed in Figure 1. These nested 
intervals can correspond not only to the modules, but also 
to the levels in the hierarchy, since proper nesting in the 
hierarchy yields a reducible control flow graph. 13 Such a 
selectively applied flow-graph reduction is roughly equiva- 
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Figure 1—Successive (left to right) interval reduction in a control flow 
graph illustrating different module correspondence, as marked by *, and by 
use of identifying letters and subscripts, such as A2 (a part of Module A) 

etc. 


lent to the reverse of a top-down application of function 
parsing guidelines. 9 

Because of this correspondence, the control flow graph 
can be used as a basis for determining the intrasystem access 
to data, given a structured programing design, and imple¬ 
mentation. With global data, the “live” access potential (L) 
for any one module is given by the union of (a) the union 
( U ) of the assignments (AN) and definitions (DN) of data 
within the module, with (b) the union of the data assignments 
and definitions available on the entrance edge of module i 
that do “reach” (R) that module: 1,2 

L i =R 1 C/(ANif/DN 1 ) (1) 

Note that the questions of whether or not the module i 
preserves or changes the definition of assignment, or 
whether or not data so specified are used in the module is 
irrelevant in determining the live access potential at any one 
module. If modules be identified in execution sequence, then 


the reach (R) expands as the control flow covers more mod¬ 
ules. Thus, the access potential increases with each module 
executed by the new assignments AN and new definitions 
DN effective in each module. 

If p indicates a count function, then the volume or amount 
of live access potential (A) in a program or system of n 
modules with global data communication is estimated by: 

A= i PU (2) 

i = l 

With passed data, the amount of live access potential is 
much smaller than for the global case with the use of struc¬ 
tured programing techniques. The live access potential is 
given by expression (1) as before, but now the set Rj is much 
reduced because it is a subset of the global data case, limited 
to the data deliberately communicated between the i th mod¬ 
ules and other modules. This also keeps expression (2) much 
smaller. 
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Figure 2—Illustration of data pass-through and pass-through chains 


With leveled data, the minimum total number of data 
items assigned values (AN) or defined (DN) is never smaller, 
and is usually larger, than the corresponding values for the 
disjoint strings. This is because the disjoint strings make 
provision only for data items actually accessed, whereas 
leveled data may include data defined or assigned for con¬ 
sistency, elegance, or symmetry of conception. 

Expressed as a ranking, with 1 for the most desirable, and 
4 for the least, passed disjoint strings are of rank 1, and 
global leveled data are of rank 4. This is summarized as the 
top row in Table I. 

Access values 

The access to data values treats each item of data as a 
different item of data when it takes on a different value. A 



Figure 3—Illustration of weak congruence between control flow marked as 
heavy lines, and data flow marked as light lines 


count of these is an indicator of the diffuseness of function 
in the hierarchy and of the difficulty of debugging and main¬ 
taining a system of program. 

The estimating procedures for access to values used for 
global data has been well presented by Allen and Cocke. 2 
Domains are difficult to specify from the flow graph and 
from the hierarchy. 10 In practice, the availability of input- 
output tables is helpful. 7 

With passed data, the access to data values is much re¬ 
stricted, and domains for data entering a node from along 
any edge of the control flow graph can be cleanly specified. 
In terms of data values used in the module, the access is the 
same as for the global case noted above. But to this must be 
added the access to values arising from the communication 
of the data through the module. Much data may have to be 
passed through some modules unchanged in value or identity 
in a hierarchical structure. Diagramed in Figure 2 is a situ¬ 
ation where an item of data is assigned an identity and value 
in module X and accessed in module Y to be modified in 
value but not identity for use in module Z. The communi¬ 
cation from X to Y involves a pass-through chain containing 
three pass-through nodes, and from Y to Z of two pass¬ 
through nodes. 

With leveled data, the count of the data values accessed 
is greater than for the disjoint string, because extraneous 
values are presented in a module at levels above the lowest 
in the hierarchy. 23 Hence, the ranking for access to values 
as summarized in Table I is best for the global disjoint string, 
and worst for the passed leveled data. 



Figure 4—Illustration of strong congruence between control flow and data 

flow 
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Congruence 

When the pattern of the data communication among the 
modules of the hierarchy has a high degree of congruence 
with the pattern of the flow of control, design, implemen¬ 
tation, and maintenance work is easier. People only have to 
know one pattern to understand two aspects of the program 
or system. The use of input-output tables helps reveal the 
degree of congruence between data and control flow. 7 

Global data offers little congruence with the nested hier¬ 
archy of the tree, as diagramed in Figure 3. This is the case 
for both leveled data and disjoint strings. This is because 
the data flow is not constrained to match the control flow, 
and in practice is usually a network, not a tree if the modules 
be treated as nodes for data flow. 14 

Passed data offers excellent congruence with the nested 
hierarchy of the tree, as diagramed in Figure 4. This is the 
case for both leveled data and disjoint strings, but the dis¬ 
joint string reduces the presence of unused data. The rank¬ 
ing, therefore for congruence places the passed disjoint 
string in the best position, and gives a tie for the two alter¬ 
natives in the global data case, as shown in Table I. 


Coupling 

While the concept of coupling comes from structured de¬ 
sign, it can be applied to software designs done with struc¬ 
tured programing. Using the definition and scale of coupling 
as laid out by Myers, the global data case is at best common 
coupled. 21 This scale position is the same for both leveled 
data and for disjoint strings. 

The passed data case requires a closer inspection. Using 
leveled data results in stamp coupling except sometimes at 
the leaf position in the hierarchy. Using disjoint strings en¬ 
ables the consistent use of elementary items of data through¬ 
out the hierarchy, and results in the more desirable data 
coupling between modules on adjacent levels vertically in 
the hierarchy. In either situation, no coupling at all char¬ 
acterizes all of the other coupling relationships between 
modules. In short, the ranking of coupling is the same as for 
congruence, as noted in Table I. 


Strength 

While the concept of strength comes from structured de¬ 
sign, it can be applied to software designs done with struc¬ 
tured programing. Using the definition and scale of strength 
as laid out by Myers, the evidence on strength is moot. 21 
Sometimes one alternative seems to go with higher strength 
and sometimes not. Hence, the ranking for strength is a tie 
for all, as shown in Table I. 

While not evidence, an observation may add insight. Since 
in structured programing, the focus is on the function of a 
module, the data flow into and from a module serves effec¬ 
tively as a definition of that function. For high strength, a 
module must require all of its input to yield its outputs, and 
not be partitionable into two or more parts with no increase 


in the number of intermodule items of data needed, ignoring 
pass-throughs. Therefore, a partitioning of functions implies 
a partitioning also of data flow if high strength is to be 
maintained, as diagramed in Figure 5. 6,8 The means of 
achieving that data flow is really an independent question. 
In a high-strength design, the actual access of data for use 
is generally closely similar whether the design or the imple¬ 
mentation be passed data or global, but the “pass-through 
unchanged” pattern is very different. 


Overhead 

Execution overhead in an implemented program or system 
depends on many factors, including the computer configu¬ 
ration, the language, the compiler, the operating system, the 
shop practices, etc. Even if attention be given only to two 
aspects, such as CPU cycles used, and amount of internal 
storage committed, only general trends can be described. 
No firm consistent evidence is available, or probably even 
possible beyond an instance by instance basis. 

Global data are the most economical of storage space and 
fastest in execution speed if disjoint strings be handled. Both 
measures are impaired by using levels of abstraction because 
of the increased size of the units of data handled. This is 
reflected in the rankings shown in Table I. 


START 

INPUTS MODULES OUTPUTS 



PARSE 


Shown in 
Chapin chart 
form 


EXPANSION 

INPUTS 



OUTPUTS 



Figure 5—Diagram of some data implications of high strength 
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Passed data are still less economical and slower, again in 
the same two steps, because of the increased handling of 
data that are involved. The higher the strength of the mod¬ 
ules, the smaller is the impairment because the more elegant 
is the use of data. But the smaller the modules the larger is 
the impairment, because calling and passing becomes a 
larger proportion of the total work to be done. This degrades 
the ranking for the passed data alternatives, as shown in 
Table I. 


Human convenience 

Both the global and the passed data are convenient for 
people, but in different ways. The global data offers single 
description (declaration), economy of effort, and ease of 
reference to the description for people. But global data also 
offer the least visibility into the steps in the use, transfor¬ 
mation, and production of the values of the data—a critical 
matter in a function-oriented approach. 

The passed data offer uniformity in conventions and high 
visibility. But the cost is in redundancy and repetition, both 
in the designs and in the code. At the compile stage the 
language conventions provide an automatic check on human 
performance. This enforcement helps keep the data visibility 
high. 

Convenience is also both aided and hurt both by leveled 
data and by disjoint strings. The leveled data provide an 
easily comprehended grouping of data but hide data identi¬ 
ties—a serious matter in debugging and maintenance. The 
disjoint strings lack the mnemonic quality that aids the 
human memory, but do focus attention on actual functions. 

A ranking of the human convenience of necessity is sub¬ 
jective, and reflects value judgments about operating envi¬ 
ronments as well. An informal survey conducted over half 
a year’s time involving persons who design computer soft¬ 
ware and try to use structured programing techniques, and 
including people from all across the continent, indicated the 
preferences shown in Table I on convenience. 


Maintainability 

Provided the visibility is good, the reports from people 
doing the work seems consistent on maintainability: the 
passed data in disjoint strings has the lowest cost of main¬ 
tenance. Some of this comes from the access measures noted 
earlier, some from congruence, and some from coupling. 
People can follow the items of data more easily, and the 
functions are usually more clearly specified and understand¬ 
able with the passed data in disjoint strings. By contrast, the 
leveled data have to be peeled away to specifics, and hence 
involve more maintenance effort. 

For these same reasons, the global data offer less aid in 
maintenance. The value changes in the data are more diffi¬ 
cult to follow, especially when leveled data are used. In the 
absence of countermeasures, it takes people longer to find 
out what is going on. Hence, in the summary ranking, the 


passing of disjoint strings is shown as a 1, and the leveled 
global data is shown as a 4. 

System quality 

System quality has many measures. One is number of 
bugs encountered over some period of time. Using tech¬ 
niques described by Haney and Myers, it is possible to give 
a quantitative tinge to measuring system quality in terms of 
bugs. 12 ' 21 

Applying those measures to examples drawn from indus¬ 
trial and governmental computer applications, indicates that 
passed data in disjoint strings has the lowest expectation of 
bugs. Passed leveled data has an expectation about one-third 
higher. Global data in either form yield an expectation for 
nearly double the number of bugs. This gives a tie in the 
ranking shown in Table I. 

Criticism of these techniques and the use of bug counts 
as quality measures has been heard. Both are admittedly 
imprecise—but they are also the best quantitative measures 
currently available. Some of the quantitative observations 
offered by Brooks are not inconsistent with the rankings 
shown in Table I for system quality. 4 


EVALUATION 

A review of the evidence indicates a strong leaning in one 
direction as the Overall summarizes in Table I. The avail¬ 
ability of this evidence gives the opportunity to take a fresh 
look at some alternatives and make rational choices to fit 
given situations and needs. Let us consider local, internal, 
and external data, for example. 

Local variables are items of data made available in the 
design and implementation to only selected modules. If they 
are accessible only in the modules in which they are declared 
or defined, then they are true internal variables. If they are 
accessible to other modules, then all of those modules must 
be in the same branch of the tree that represents the nested 
hierarchical relationship among the functions. Effectively, 
the locals thus used become “restricted” globals. Local 
variables are well regarded in the theory of structured pro¬ 
graming. 19,27 

Used with global data, local data are a step in the direction 
of passed data. Used with passed data, local data offer a 
relaxation of the visibility enforced with passed data. But 
the use of true internals degrades neither passed data nor 
global data usage. Internal data use may even enhance either 
but especially global data. 

“External” data are another possibility. These are items 
of data communicated between specific modules anywhere 
in the hierarchy—not just between immediate superiors and 
subordinates. It is like a private line for communication 
external to and addition to those provided by the passing of 
data. 

External data are better regarded in structured design than 
global data because they are more visible to people. 21 But 
their flow does not follow the tree, and as they are usually 
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TABLE I.—The Most Favorable Rank is 1, the Least 
Favorable is 4. 


SYNOPSIS OF RANKING 

Evidence Passed Data Global Data 

Factor 

Leveled 

String 

Leveled 

String 

Volume 

2 

1 

4 

3 

Values 

4 

3 

2 

1 

Congruence 

2 

1 

3.5 

3.5 

Coupling 

2 

1 

3.5 

3.5 

Strength 

2.5 

2.5 

2.5 

2.5 

Overhead 

4 

3 

2 

1 

Convenience 

2 

1 

4 

3 

Maintainability 

2 

1 

4 

3 

Quality 

2 

1 

3.5 

3.5 

OVERALL 

2 

1 

4 

3 


used, they provide another (sometimes complicating) data 
flow for people to understand simultaneously with either 
passed or global data. A common use of external data ac¬ 
cessibility is the use of scratch files to communicate data 
between selected modules, often relatively high in the tree. 

Is there some form of intrasystem data communication 
that would rate better than any of those discussed thus far? 
Consider the following possibility, termed “monitored 
data,’’ that combines the strong features of the forms and 
varieties already discussed. 

1. Define or declare with a unique name each and every 
data item only once in the software system to be created. 
Incorporate in the definitions or declarations any list-linking 
or component or group-element relationships among the 
items of data. 

2. Enable by a specific visible means in the documenta¬ 
tion for the module, each module that may access the value 
of any given item of data or may assign the value, on an 
item by item basis, noting the role of the item of data. The 
roles may be for control, for processing, for modification of 
value, or for communication through unchanged. 

3. Record and associate with the definition or declaration, 
the identification of the accessing and assigning modules and 
the roles of the data in those modules. This documentation, 
while not intrinsic to monitored data, adds substantially to 
the human convenience and to the maintainability. 

4. Disenable all other modules for all items of data by a 
means that prevents their implementation if any access is 
explicitly or implicitly attempted in or by them to any item 
of data for which they have not been specifically enabled. 

5. Make any access to any item of data be to all parts of 
that item of data (for example, enabling for a 1000-byte 
record yet only using a 2 byte field in it would be illegal) 
except where explicit decisions result in alternative access. 

Some comparison of monitored data with the forms and 
varieties considered earlier may be interesting (see Table I). 
For systems and programs involving more than about 50 


intermodule items of data, on volume, congruence, coupling, 
and strength, the monitored data alternative ranks the same 
as the passed-string alternative. On values and overhead, it 
ranks the same as the global-string alternative. On system 
quality, it ranks better than the passed-string alternative, by 
close to a factor of two. For systems and programs of smaller 
size and for some strongly mathematics oriented systems, 
the passed-string alternatives appear to be superior to the 
use of monitored data. But from the limited experience thus 
far, for most non-trivial systems and programs implemented 
in COBOL, FORTRAN, PL/1, JOVIAL, and BASIC, the 
monitored data appear to be the superior alternative. 

Insufficient experience is available to give a confident 
ranking on convenience and maintainability. Some obser¬ 
vations may be indicative, however. On convenience, the 
monitored data appear to offer all the advantages of both 
the passed and the global alternatives, and avoid the low- 
visibility of the global and some of the redundancy of the 
passed data alternatives. The verification of the enabling in 
points # 2 and # 4 requires either a manual or a software 
step. An automatic software step improves the human con¬ 
venience. The usefulness and precedent for such steps al¬ 
ready exists in some structured programing practices using 
precompilers, data base administrators, librarians, and proj¬ 
ect files. 3 Input-output tables can also be a useful aid. 7 
Because of the excellent visibility it produces, the monitored 
data alternative appears to offer a maintainability potential 
superior to that of the passed-string alternative. 


CONCLUSION 

Among the four alternatives considered, the weight of the 
evidence favors the passed form of intrasystem data com¬ 
munication, especially where the overhead is not of serious 
concern. The disjoint string ranked superior or equal to the 
use of data in levels of abstraction on all nine factors con¬ 
sidered. Except on convenience and maintainability where 
little evidence exists yet, the monitored form of intrasystem 
data communication appeared to be superior or equal to the 
passed-string alternative. 
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INTRODUCTION 

There is currently interest in measuring the complexity of 
a computer program with the evaluation of a program’s 
control structure via its flowgraph representation. 1,2 How¬ 
ever, the inclusion of the effects of data on these measures 
is often lacking. This paper proposes a new measure of 
program complexity that attempts to identify regions of lo¬ 
cality by combining a control structure approach with infor¬ 
mation about data usage to get a valid measure of overall 
program complexity. The need for objective measures of 
program structure is important if the area of programming 
language design and implementation is to take on a more 
formal basis from the ad hoc techniques currently used. 

This work is related to that of Hellerman, 3 Savage, 4,5 and 
Chaitin 6 in that the complexity of an algorithm is defined as 
the number of bits in the algorithm’s representation, i.e., 
the number of bits in a program implementing the algorithm. 
A program implementing a given algorithm is considered to 
be better than another program implementing the same al¬ 
gorithm if it has fewer bits in its representation. In order to 
provide a standardized representation for algorithms, the 
concept of a Hierarchical Abstract Computer is defined 
(HAC). Based on a primitive-level HAC, hierarchical levels 
of HAC modules are used to specify the structures and 
sequences of a specific algorithm. The modules are chosen 
according to the concept of a Prime Program Parse, 7 which 
defines a particular, unique sequence of subroutine decom¬ 
positions from an original flowgraph representation of an 
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algorithm. The complexity of the total program is defined to 
be the sum of complexities of each of the modules used to 
form the hierarchy. 

The difficulty of writing program code to implement an 
algorithm has been related to the amount of information that 
must be manipulated in the process of writing the program. 
There have been many attempts recently to simplify the 
process, most conspicuously aimed at the control structure 
complexities. Structured programming has been expounded 
as a method by which programs can be written with mini¬ 
mum difficulty, and mainly involves restraints on the use of 
control structures. There are also some methods which are 
directed at data structure complexities, such as the concepts 
of data abstraction and information hiding. These methods 
are concerned with the simplification of the data that is 
manipulated by the program code. By reducing the data 
available for referencing to only that which is directly 
needed by a section of code, the possibility for confusion is 
reduced. As much as possible, references to data are main¬ 
tained in small sections of code, and are kept exclusive, in 
that only those sections are allowed to act on the data. In 
this manner, local data responsibilities are easy to isolate 
and to maintain. These local responsibilities are known as 
clusters, and their exclusiveness and privacy is known as 
information hiding. 

It is assumed that a program is well-structured if clusters 
of data and control activity are small. The resultant small 
subroutines are thus accompanied by correspondingly small 
data spaces. The purpose of this research is to be able to 
measure different implementations of the same algorithm so 
as to determine the particular implementation which is best 
in the sense of being well-structured or least complex. The 
next section discusses some relationships between an algo- 
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rithm and the set of programs which implement the algo¬ 
rithm. 


ALGORITHMS AND PROGRAMS 

An algorithm has been defined to be a finite set of rules 
which specify a sequence of operations used to solve some 
specific problem. The operations specified by the rules are 
selected from a set ft of operators used by the algorithm. 
The proper sequencing of these operations depends on the 
state of the data items used by the algorithm, and is de¬ 
scribed with the use of a meta-algorithm or program which 
directly implements the algorithm. The essential difference 
between an algorithm and its implementing program is the 
specification of the operation sequencing through the use of 
control statements. Each algorithm has an infinite number 
of programs which implement it, some of which are consid¬ 
ered to be well-structured, or less complex. It is this class 
of programs which a complexity measure attempts to iden¬ 
tify. 

Algorithms are rarely completely specified, in that as¬ 
sumptions are made about the characteristics of the data 
types and the operators which manipulate the data. Yet each 
operator is itself described by an algorithm at a lower level, 
and so on, down to some indivisible, lowest level set of 
operators and data types. Computer programs are written in 
high-level languages whose operators are implemented in 
machine languages, whose operators are in turn imple¬ 
mented in micro-code, whose operators are implemented in 
hardware gates, etc. The next section proposes a primitive 
set of operators and data types to be used as a basis upon 
which complete specifications of algorithms can be hierar¬ 
chically built. 

Rarely, however, does the operator set of the algorithm 
match the operator set of the programming language. A 
central problem in programming is identifying and building 
the inherent operators and data types that make up the 
algorithm, using the available, lower-level tools. Usually, 
the algorithm description given to the programmer does itself 
not adequately recognize the structures within the algorithm 
specified, and so, severely complicates the problem facing 
the programmer. His task is to perform certain transforma¬ 
tions to the algorithm specification without altering its basic 
structure so as to implement the request as simply as pos¬ 
sible. 

Let A be the set of data items used by an algorithm or 
program, and let D be a subset of A. The Execution Se¬ 
quence of an algorithm relative to D is defined to be the 
sequence of states assumed by the members of D. 

Two programs are execution equivalent relative to D if 
they have the same execution sequence for all inputs. Dif¬ 
ferent programs can be execution equivalent because the 
occurrence of control operations is ignored, having no effect 
on the execution sequence. 

Two programs are functionally equivalent if, given the 
same input values, they produce the same output values. 

Two programs are semi-execution equivalent relative to 
D if, for each d E D, the programs are execution equivalent 


relative to d. That is, each data item of interest takes on the 
same set of values, but not necessarily all in the same rel¬ 
ative order as the other data items. 

It is a trivial result that if two programs are semi-execution 
equivalent, then they are also functionally equivalent. Func¬ 
tional equivalence is the weakest restriction, in that there 
need be no similarity in the two algorithms used by the 
programs. Execution equivalence is the strongest restric¬ 
tion, requiring two programs to have identical data trans¬ 
formations. The restriction of semi-execution equivalence is 
felt to be a reasonable criteria for comparing different pro¬ 
grams, in that the implemented algorithms are not only func¬ 
tionally equivalent, but also structurally similar, in that all 
of their data items are undergoing similar transformations. 
There is a certain latitude available to the programmer in 
this situation to strive towards a simple implementation. 

HIERARCHICAL ABSTRACT COMPUTERS 

The concept of a set of programs interacting at different 
levels, and working together to implement an algorithm, 
leads naturally to the following definition of a Hierarchical 
Abstract Computer (HAC). 

Def: A Hierarchical Abstract Computer (HAC) is a quad¬ 
ruple (A,/, fl ,T) where A is a set of distinct named data 
items, each of which is associated with a member of T, 
the set of legal data types, I is a set of named instruc¬ 
tions, and H is a finite set of operation codes. Each 
instruction is of the form: 

(Ddydz . . . dnXyXz . . . Xm 

where wE Cl, diE A and jq is the name of an instruction 
in /, or jq=0. The df s are the data arguments for the 
operator co, and the Xj’s represent the possible alter¬ 
natives for the next instruction to be executed. The data 
items and the instruction addresses (names) are num¬ 
bered beginning with 0, and a transfer to address 0 means 
to halt execution for that HAC. 

Operation codes are either composite or primitive. A com¬ 
posite code is itself implemented by a lower-level HAC, 
carrying out some specific algorithm on the input data. A 
primitive operation code is, in some sense, indivisible, and 
is defined to be one of the three operators: 

TEST d x t X 2 

SET d x 1 

CLEAR d x 1 

where dE A, and is associated with the data type BIT, having 
the value 0 or 1, and x 1 ,x 2 el. These instructions have the 
semantics: 

TEST If the contents of d= 0, then the next instruction 

is to be found at address x t ; otherwise, the next 
instruction is at x ^. 

SET Set the data item d equal to 1. 

CLEAR Set the data item d equal to 0. 

These three codes form the Primitive Basis, Cl p , and, if 
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a HAC consists of together with the data type J={BIT}, 
then the HAC is said to be a Primitive HAC, or PH AC. An 
important characteristic of £l p is that it is complete, that is, 
it can compute all finite functions /: {0,l} n —>{0,1}. This re¬ 
sult follows directly from the PHAC implementation of the 
basic Boolean functions AND, OR, and NOT. 

As described, a HAC program can be decomposed into 
lower and lower level modules until only Primitive HACs 
are obtained, thus forming a tree of modules. Conversely, 
given a PHAC implementation of an algorithm, sections of 
code can be removed and replaced by a single invocation to 
a new composite operator, with as many parameters as there 
were global variables across the code segment. 

While a HAC is only as powerful as a finite state auto¬ 
mata, it is still sufficiently powerful to model real computer 
programs that do not use auxiliary storage, and, unlike finite 
state models, can easily be measured with a complexity 
measure. This model can, however, be easily extended to 
include tapes. As in real hardware, specific addresses for 
data, control, and status registers can be defined resulting 
in a memory mapped I/O. Placing information into these 
registers causes information to be read or written from a 
tape and to appear in the data register. Thus storage may 
actually be infinite, but accessible only via a finite set of 
specific addresses. 

PROGRAM COMPLEXITY 

Given a HAC implementation of an algorithm, it is now 
necessary to define a complexity measure on this set of 
modules. The complexity of a HAC H is defined to be the 
sum of the complexities of each of the instructions in the 
HAC: 

CiH)= CO) 

Let L A =log 2 | A |=number of bits needed to address the 
data space, 

L/=log 2 | / |+ l=number of bits needed to address the 
instruction space I (with the addition of address 
zero to be used as a STOP instruction), and 
L n =log 2 | fl |=number of bits needed to specify the 
operation code. 

Then, the complexity of an instruction i with m data ar¬ 
guments and n addresses is: 

C(i)=Ln+m*L A +n*Li 

For PHAC programs, the complexity of each TEST in¬ 
struction is (1.58 +L a +2*Li) and the complexity of each SET 
and CLEAR instruction is (1.58 +L a +Li). 

Thus, the complexity of a HAC implementation of an 
algorithm is defined to be the sum of the complexities of 
each of the HAC modules which form the implementation. 
For each program, however, there are many possible mod¬ 
ularizations, any one of which could be chosen by the pro¬ 
grammer, but which may not reflect the structure of the 
program itself. It is desirable to have a canonical modular¬ 


ization for a program to avoid this problem. This represen¬ 
tation should be unique for a given program, and should 
correlate closely with the structural properties of the pro¬ 
gram. The Prime Program Parse meets these criteria. 7 

A proper program is a flowgraph with a single input arc, 
a single output arc, and with each node in the flowgraph 
being on a path from the input arc to the output arc. A prime 
program is recursively defined as a proper program with no 
proper prime subprograms of greater than one node. The 
process of identifying the hierarchy of prime programs which 
make up a compound program is known as a prime program 
parse. Such a parse is unique up to associativity of se¬ 
quences of statements. 

PROPERTIES OF MEASURE 

This measure has been used to measure the complexity of 
several programs implementing common algorithms. In one 
example, an 8-bit adder was programmed on a primitive 
HAC, using 78 instructions, and resulting in a complexity of 
918.26. The prime program parse of the program yielded 8 
subroutines, all invoked by a single controlling routine. This 
program was restructured by forming three composite op¬ 
erators, resulting in a complexity of 471.76. This savings in 



Figure 1—Unstructured version of program 
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complexity was due primarily to the recognition that 6 of 
the 8 modules in the prime program parse performed the 
same function, and thus represented a large degree of re¬ 
dundancy. The two programs are execution equivalent. 

As an example of the comparison of two programs which 
are semi-execution equivalent, consider those shown in Fig- 



Figure 2—Structured version of program 


ures 1 and 2. Figure 1 illustrates a nonstructured program 
which has a complexity of 53.16, as determined from the 
prime-program parse. Figure 2 illustrates a structured pro¬ 
gram, having a complexity of 52.02, and consisting entirely 
of small (size 1 or 2) prime programs. 

The second example illustrates a basic result, namely that 
for HAC programs which consist of two prime programs 
with all variables being global, the minimum complexity 
occurs when the number of instructions is equally divided 
between the two subprograms. This result is obtained di¬ 
rectly from a minimization of the expression for the com¬ 
plexity of such a program. Repeated applications of this 
result yield the conclusion that the minimal complexity for 
a program occurs when only a structured basis for the con¬ 
trol is used, i.e., only the control structures sequence, if- 
then-else, do-while, repeat-until, do-while-do, if-then, and 
function. 

This proposed measure has a demonstrated sensitivity to 
both control and data structuring within programs, and, be¬ 
cause of its close relationship to the structured programming 
control graphs and prime programs, is felt to be a valuable 
tool in the quantification of overall complexity in the effort 
to formalize the intuitive concept of a good program. 
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INTRODUCTION 

The execution of software test cases and the verification of 
test results may be performed automatically by a new type 
of program called an automatic software test driver. When 
using an automatic test driver, a formal test procedure is 
coded in a special test language. The test procedure takes 
the place of the test data and test setup instructions of 
conventional testing, and controls the automatic test driver. 
An automatic software test driver applies one test procedure 
to all or part of a target program, executes all of the test 
cases specified in the test procedure, and verifies that the 
results of each test case are correct. This paper describes 
the Fortran Test Procedure Language (TPL/F) which was 
developed at General Electric and is used for specifying test 
procedures for Fortran software. 1 

The concept of automatic software test drivers is a new 
idea that has been evolving slowly over the past six years 
and just now seems to be approaching the point where soon 
it may play a significant role in the development and main¬ 
tenance of production software. Two other automatic soft¬ 
ware test drivers that were developed independently in re¬ 
cent years and have their own software test languages are 
described in References 2 and 3. 

The Fortran Test Procedure system is illustrated in Figure 
1. One test procedure coded in TPL/F and the source code 
for one or more modules of the target program are processed 
by the automatic test driver which executes all of the test 
cases specified in the test procedure, and produces a brief 
test execution report (Figures 2 and 3) stating which test 
cases failed, if any, and the degree of testing coverage ac¬ 
tually achieved by the test procedure. 

Test cases consist of input data for the target program and 
model output data. The automatic test driver actually exe¬ 
cutes the target program for each test case, feeding the input 
data to the target program and comparing outputs from the 
target program with the model outputs specified in the test 
procedure. Incorrect outputs produce a diagnostic in the test 
execution report (Figure 3). 

Since the TPL/F system has access to the target program’s 
source code, it can monitor which statements were actually 
executed and which branches were actually traversed while 
executing a test procedure. This type of information pro¬ 


vides valuable feedback to the test designer and is generally 
not available when testing is done manually. 

The specific goals of the TPL/F automatic software test 
driver are as follows. The need for writing drivers and stubs 
for module and subsystem testing is eliminated since the 
TPL/F system can test any combination of one or more 
modules independently of the rest of the target program. 
The TPL/F test language provides a standard format for 
specifying software tests and the test procedure processor 
provides a standard test execution setup. Since formal test 
procedures specify the correct outcomes of test cases, the 
test procedure processor automates the verification of test 
execution results. 

TPL/F TEST PROCEDURES—AN OVERVIEW 

TPL/F test procedures and the major elements of the TPL/F 
language are introduced in this section by means of an ex¬ 
ample. The following section describes the TPL/F language 
in greater detail. For the present example, we show a simple 
test procedure for the Fortran target module of Figure 4. 
The target module, IOSUB, is a subroutine and needs to be 
called by a higher level module for execution. IOSUB also 
invokes a lower level subroutine, LABEL, which is not 
under test and must be simulated when testing IOSUB. On 
each entry to IOSUB, one unformatted record is read from 
logical unit 12 and LABEL is called once. 

A simple test procedure to test IOSUB is illustrated in 
Figure 5. The test procedure contains three test cases, each 
of which causes an actual execution of the target module, 
IOSUB. The MODULES statement names the target mod¬ 
ules (IOSUB of Figure 4) under test. 

TPL/F test cases are executed in three steps. First, ini¬ 
tialization code is executed and puts the target program in 
a known initial state. In the example, the first two statements 
of each test case are used to initialize the target program. In 
the first test case, for example, they assign the values 1 and 
0 to the variables N1 and N2, respectively, of IOSUB. 

Second, the target program is executed. The form of the 
EXECUTE statement used in Figure 5 causes execution to 
begin at the first executable statement of IOSUB and ter¬ 
minate the first time a STOP or RETURN statement is 
encountered in IOSUB. 
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Figure 1—The TPL/F automatic software test driver 


Third, after the test case execution has terminated, an 
assertion about the final state of IOSUB is verified. In the 
first test case of Figure 5, the variables N1 and N2 of IOSUB 
are verified to have the values of 6 and 0, respectively. 

In TPL/F, stub versions of missing subprograms called by 
the target program are coded as Fortran-like subroutines 
and functions embedded in the test procedure. In Figure 5, 
for example, a stub version of subroutine LABEL begins at 
the SUBROUTINE statement and runs thru the following 
END statement. #TEST is an integer system variable whose 
value is always the index of the test case currently execut¬ 
ing. The stub version of LABEL verifies that the argument 
passed by IOSUB when executing under the first test case 
is 10 and returns the value 0 for the argument on return. 
Should LABEL call IOSUB with an incorrect argument 
value, the ABORT statement would terminate the current 
test case (but not the test procedure) and put a diagnostic 
message in the test execution report. 

The six statements beginning at the I/O SIMULATOR 
statement and continuing through the following END state¬ 
ment specify an I/O simulator for logical unit 12 of the target 
program. An I/O simulator can supply input data to the 


VERIFY FAILURE IN TEST CASE 2 AFTER TERMINATION 
(IOSUB:Nl .EQ. 0 .AND. IOSUB:N2 .EQ. 1) 

STATEMENTS EXECUTED: 100% 

BRANCHES TRAVERSED: 100% 

Figure 3—A failed test procedure execution report 


target program in response to READ statements or verify 
outputs from WRITE statements in the target program. The 
I/O simulator of Figure 5 specifies three unformatted records 
containing five integer values each, followed by an end-of- 
file mark. The first three records read by the target program 
from logical unit 12 will be taken from the I/O simulator. An 
attempt to read a fourth record would produce an EOF 
return in the target program. 

Although the test procedure in this example contains only 
three test cases, most test procedures contain a much larger 
number. Typically, a test procedure for a single Fortran 
target module of 50 to 100 statements may contain 20 to 50 
test cases, depending on the complexity of the target mod¬ 
ule’s control logic. Therefore, a more compact notation usu¬ 
ally is required to represent test cases. TPL/F uses a built- 
in macro processor which allows the general form of a test 
case to be specified as a macro prototype and each specific 
test case is represented by a single macro call. In practice, 
test procedures tend to contain only a small number of forms 
of test cases (often only one) which are invoked many times 
with different specific data values. Figure 6 shows the three 
test cases of Figure 5 as they would appear when coded as 
TPL/F macro calls. 

THE TPL/F TEST LANGUAGE 

The previous section introduced the control structure and 
the four major components of TPL/F test procedures (test 
cases, subprograms, I/O simulators, and macros). This sec¬ 
tion contains an informal and intuitive introduction to all of 
the constructs of the TPL/F test language. Complete de¬ 
scriptions of the TPL/F language and the Fortran test pro¬ 
cedure system are contained in References 4 and 5. 

Figures 7 thru 13 describe the syntax of TPL/F by means 
of syntax graphs. Rounded boxes represent terminal sym¬ 
bols of the language. Rectangular boxes are non-terminal 
symbols requiring further definition. The following Fortran 
syntactic forms (marked with an asterisk in the syntax 


SUBROUTINE IOSUB(Nl,N2) 

CALL LABEL(ICOUNT) 


ALL TEST CASES SUCCEEDED READ (12) (K(I),I-1,5) 

STATEMENTS EXECUTED: 100% 

BRANCHES TRAVERSED: 100% END 


Figure 2—A successful test procedure execution report 


Figure 4—A sample target module 
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MODULES IOSUB 

I/O SIMULATOR 12 
RECORD (1)0, 5, 1,2, 7 
RECORD (2) 6, 1, 7, 3, 4 
RECORD (3) 2, 9, 8, 0, 0 
RECORD (4) #EOF 
END 

SUBROUTINE LABEL(I) 

INTEGER IN(3),IOUT(3) 

DATA IN/10,20,0/, IOUT/0,0,1/ 

IF (I .NE. IN(#TEST)) ABORT 
I = IOUT(#TEST) 

RETURN 

END 

IOSUB:Nl = 1 
IOSUB:N2 = 0 

VERIFY (IOSUB:Nl .EQ. 6 .AND. IOSUB:N2 .EQ. 0) 
EXECUTE 

IOSUB:Nl = 0 
IOSUB:N2 = 100 

VERIFY (IOSUB:N1 .EQ. 0 .AND. IOSUB:N2 .EQ. 1) 
EXECUTE 

IOSUB:Nl = 500 
IOSUB:N2 = 501 

VERIFY (IOSUB:N1 .EQ. 1 .AND. IOSUB:N2 .EQ. 1) 
EXECUTE 

FIN 


Figure 5—IOSUB test procedure coded in the TPL/F test language 


graphs) are embedded in TPL/F. Their grammer depends on 
the syntax of the host Fortran dialect. 


identifier 

label 

integer constant 
string 

relational operator 
subprogram statement 


declaration statement 

format 

I/O list 

data list 

logical expression 


The symbol “Ftn statements ” refers to any statement of 
the host Fortran dialect. 


Test procedures 

Test cases are the primary element of the control structure 
of a test procedure (Figure 7). Executing a test procedure 


MACRO MAC(P1,P2,P3,P4) 

IOSUB:Nl = PI 
IOSUB:N2 = P2 

VERIFY (IOSUB:Nl .EQ. P3 .AND. IOSUB:N2 .EQ. P4) 

EXECUTE 

MEND 

MAC(1,0,6,0) 

MAC(0,100,0,1) 

MAC(500,501,1,1) 

Figure 6—The three test cases of Figure 5 recorded using a TPL/F macro 
definition 


consists of executing each of its test cases. Subprograms 
and I/O simulators defined in the test procedure only influ¬ 
ence the performance of test cases. Fortran declaration 
statements may be included in test procedures to define 
COMMON blocks whose variables are referenced by the 
test procedure. 

The MODULES statement names all of the target modules 
affected by the test procedure. The first name on the MOD¬ 
ULES statement is the primary target module. All test pro¬ 
cedure references to target program variables or statement 
labels, not qualified by an explicit target module name, are 
directed to the primary target module. 


Test cases 

Each test case (Figure 8) in a test procedure causes an 
actual execution of the target program. 

The EXECUTE statement specifies where to begin and 
where to terminate a test case execution, and how many 
times to execute it. For example, the following statement 
would cause its test case to be executed three times, each 
time beginning at statement label 50 of the primary target 
module and stopping whenever either statement label 25 of 
target module SUB2 or statement label 35 of target module 
SUB3 is executed 

EXECUTED FROM :50 TO SUB2:25,SUB3:35 

An exclamation mark preceding a termination label refer¬ 
ence means that the test case is to terminate without exe¬ 
cuting that statement. Otherwise, it would be terminated 
immediately afterward. The beginning statement, when 
specified, is always executed, regardless whether or not its 
reference is preceded by an exclamation mark. If no begin¬ 
ning label is specified, execution starts at the first executable 
statement of the primary target module. Regardless whether 
or not termination labels are specified, test cases always 
terminate whenever a STOP statement or a RETURN state¬ 
ment in the primary target module is executed. 

All test case statements other than EXECUTE, VERIFY, 
and USE are executed immediately prior to the start of a 
test case execution. These are usually modified Fortran 
statements (see below) supplying initial values to the target 
program. 

The VERIFY statement specifies an assertion about the 
target program. The AT clause, if present, specifies when, 
during the test case execution, the assertion is to be verified. 
The following statement causes an assertion to be verified 
whenever statement label 100 in SUB1 is executed, as well 
as immediately after the test case terminates. 

VERIFY AT SUB1:100,#TERM (SUB1:N .EQ. 0) 

An exclamation mark preceding an AT clause label reference 
causes the assertion to be verified immediately before exe¬ 
cuting the specified statement. Otherwise, verification oc¬ 
curs after executing the referenced statement. When no AT 
clause is specified, the assertion is verified after the test 
case terminates. An attempt to verify a false assertion ter- 
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MOOULES statement 


■» C MODULES > 



Identifier» 



FIN statement 


- K PIN ) 


Figure 7—Test procedure syntax 


minates the current test case (but not the test procedure) 
and causes an error message to appear in the test execution 
report as in Figure 3. 

The USE statement modifies the mapping of target pro¬ 
gram I/O references to I/O simulators and affects only the 
test case in which it appears. For example, if a test proce¬ 
dure contains I/O simulators for logical units 20 and 30, all 
target program I/O references to these logical unit numbers 
would ordinarily invoke I/O simulators for those units in the 
test procedure. The following statements appearing in a par¬ 
ticular test case, however, would direct all target program 
references to logical unit 20 to I/O simulator 30; and all 
target program references to logical unit 30 to an actual 
logical unit of that number. 

USE 30 FOR 20 

USE #ACTUAL FOR 30 

The #SCRATCH clause directs target program I/O opera¬ 
tions to a temporary scratch file. 

Subprograms 

Fortran subroutines and functions defined in a test pro¬ 
cedure (Figure 9) may be referenced inside the test proce¬ 
dure or may be used as subprogram stubs. 

When referenced inside a test procedure, subroutines may 
be used to perform complicated recurrent test case initiali¬ 
zation tasks or Boolean functions may be used to verify 
complicated assertions about the target program. When used 
as subprogram stubs, subprograms defined in the test pro¬ 
cedure are surrogates for missing subprograms called by the 
target program. Any subprogram defined in a test procedure 
can be used in either manner. 


I/O simulators 

I/O simulators are used to simulate I/O devices and files 
referenced by the target program. An I/O simulator (Figure 
10) is a closed subroutine and is entered each time its logical 
unit number is referenced by a target program I/O statement. 

Each I/O simulator has a unique record counter which 
always points to the current record on the simulated I/O 
unit. Prior to each test case execution, all record counters 
are reset to zero. While executing a test case, each time the 
target program reads or writes the simulated unit, its record 
counter is incremented. 

I/O simulators can simulate either random or sequential 
devices. The NO RESET clause on the I/O SIMULATOR 
statement preserves the record counter value from one test 
case to another. The NO REWIND clause inhibits target 
program REWIND statements on the I/O simulator. Ex¬ 
amples: 

I/O SIMULATOR 12 RANDOM(IOOO) 

I/O SIMULATOR 16 NO RESET 

The first statement defines a random I/O simulator for logical 
unit 12 with a maximum record index of 1000. The second 
statement defines a sequential I/O simulator for logical unit 
16 whose record counter is not reset between test case 
executions. The latter form is used for providing multiple 
test cases with different data files from the same logical unit. 

The RECORD statement is the primary I/O simulator 
statement for simulating I/O devices and files. When the 
target program executes a READ or WRITE statement on 
a simulated logical unit, the target program is interrupted 
and control passes to the I/O simulator. The I/O simulator 
executes until an operable RECORD statement is encoun- 
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Figure 8—Test case syntax 


, subprogram statement * 


Ftn statement * 


END statement 


msm: 


Figure 9—Subprogram syntax 
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I/O SIMULATOR 
statement 


<|/Q S1MULATQR>—,,1- 


RANDOM integer constant 


[» C NO REWIND > 4 


NO RESET 






tered. If the I/O simulator was invoked by a target program 
READ statement, input data values from the RECORD 
statement are passed to the target program READ list and 
control is returned to the target program. If the I/O simulator 
was invoked by a target program WRITE statement, the 
data output by the WRITE statement is captured and com¬ 
pared with data in the RECORD statement. If the compar¬ 
ison is satisfied, control returns to the target program; oth¬ 
erwise an error message is written in the test execution 
report. 

RECORD statements that specify a record range are op¬ 
erable only when the value of the host I/O simulator’s record 
counter falls within the specified range. Otherwise, the REC¬ 
ORD statement is treated as a no-op. A record range may 


be specified as a single record number as in 
RECORD (5) . . . 

or as a minimum and maximum value as in 
RECORD(5:10) . . . 

The first statement is operable only for the fifth record on 
the simulated device and the second statement for the fifth 
thru tenth. A RECORD statement with no record range is 
always operable. 

A RECORD statement may describe either formatted or 
unformatted records, depending on whether or not a format 
reference, indicated by a #, is specified, as in the following 
examples. 













external identifier 


identifier« 


statement reference 







V 1 




l/O SIMULATOR 
statement 




__ 11 __ 

I USE statement | 


| RECORD statement | 

(RETRIEVE statement! 


VERIFY statement] 

[EXECUTEjitatementl 

If-*- 

Ftn statement * 




_ 




| END statement | 



L. 


L. 

L. 



MEND statement 


[fMSM 
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conditional 


-^ tCOND statement 


% 


ELSE statement 


7 


ENDC statement I 


COND statement 


» (CONP] 


l »| string term |— 

relational 

| >| string term | ? 

1 


f 

'-^numeric term) ^ 

relational 
operator * 

[—^numeric term|- ' 




ELSE statement 


ELSE ~) -► 


ENDC statement . ... . 

- -» < ENDC ) -► 


string term 


-K> 


I 


string» [ - 


Identifier ★ | 


J 


<D—► 


r ~*C «NARG 


numeric term 


Integer 
constant • 


ititler 3-KD-J 


'-•C -VAL ) »( D fc p dentilier i 

Figure 13—Conditional construct syntax 


RECORD #(315), 10,20,30 
RECORD #999, (X(I), 1=1,10) 

RECORD 7,12,(18,1=1,10) 

The first two statements simulate formatted records and the 
third, an unformatted record. Formatted records must be 
generated in response to target program formatted I/O state¬ 
ments and unformatted records must be generated in re¬ 
sponse to target program unformatted I/O statements. 

The #ANY clause in a RECORD statement verifies that 
the I/O simulator was invoked by any target program write 
operation. The #NOLIST clause verifies that the I/O sim¬ 
ulator was invoked by a formatted write operation with no 
I/O list and is used primarily for skipping titles and format¬ 
ting information when verifying formatted output records. 

The #ERR clause in a RECORD statement simulates an 
I/O error and causes the ERR=label exit to be taken from 
the corresponding target program I/O statement. The #END 
clause simulates an end of file condition and is used to verify 
a target program ENDFILE statement or cause the END- 
=label exit to be taken from a target program READ state¬ 
ment. 

The RETRIEVE statement is used for verifying complex 
assertions about target program outputs. When the host I/O 
simulator is invoked by a WRITE or PRINT statement, the 
output record is captured, optionally interpreted according 
to a format, and assigned to variables in an I/O list. While 
the RECORD statement is limited to simple tests of equality 
between output variables and known values, the RE¬ 


TRIEVE statement makes output records available for any 
type of test. The first example of the following section (EX¬ 
AMPLES) illustrates the use of tho RETRIEVE statement. 

TPL/F modifications to embedded Fortran statements 

Fortran statements may appear throughout test proce¬ 
dures. The initialization part of a test case definition, for 
example, usually consists of Fortran code only. When ap¬ 
pearing inside a test procedure, any Fortran statement may 
be modified as follows. 

External identifiers, statement reference counters, and 
system variables (Figure 11) may be referenced wherever a 
variable would usually appear in a Fortran statement. 

An external identifier refers to a variable or array in the 
target program. The initialization code and assertions in the 
example of the previous section contained external identi¬ 
fiers pointing to the variables N1 and N2 of target module 
IOSUB. 

Statement reference counters provide a means for veri¬ 
fying the number of times a target program statement is 
executed. Every test procedure reference to a statement 
reference counter causes an integer-valued counter to be 
associated with the indicated statement in the target pro¬ 
gram. Prior to each test case execution, all statement ref¬ 
erence counters are reset to zero. During test case execu¬ 
tions, statement reference counters are incremented each 
time their associated statements are executed. The following 
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statement, for example, verifies that the statement at label 
100 of target module SUB1 was executed exactly five times 
during the current test case execution. 

VERIFY (#SUB 1:100 .EQ. 5) 

Three special symbols are used to reference system var¬ 
iables whose values are controlled automatically by the test 
procedure processor. 

The value of #TEST is the index (e.g., first, second, etc.) 
of the currently executing test case. 

#REF is defined as follows. Every subprogram and I/O 
simulator has a reference counter associated with it. All 
reference counters are reset to zero prior to each test case 
execution and incremented by one on each entry to their 
associated subprograms. Inside a subprogram or I/O simu¬ 
lator, #REF has the value of the subprogram or I/O simu¬ 
lator’s reference counter. In all other places, #REF is al¬ 
ways equal to zero. 

Inside an I/O simulator routine, #RECORD has the value 
of the I/O simulator’s record counter. Elsewhere, the value 
of #RECORD is always zero. Examples: 

Y=VAL(#REF,#TEST) 

IF (#RECORD .EQ. N) GO TO 100 

Ordinarily, Fortran I/O statements embedded in a test 
procedure are not affected by I/O simulators. A logical unit 
designator prefixed by a pound sign in an embedded Fortran 
I/O statement temporarily suspends this rule and causes the 
I/O statement to be treated the same as a target program 
reference to the logical unit. For example, the following 
statement appearing inside a test procedure references the 
I/O simulator for logical unit 10, not the actual logical unit 
10 . 

READ (#10) X,Y,Z 

The ABORT statement is used inside subprograms and 
I/O simulators to terminate the current test case when an 
error is detected. Terminal assertions of aborted test cases 
are not verified. 


MACRO MAC(N,ALPH) 
COND (#NARG .EQ. 1) 

:A = 0. 

COND(#VAL(N) .NE. 0) 

:B = 0. 

ENDC 
ELSE 
:C = 0. 

COND (ALPH .EQ. ‘ABC’) 
:D = 0. 

ENDC 

ENDC 

MEND 

Macro Call 
MAC(0) 

MAC(2) 

MAC(2,XYZ) 

MAC(2,ABC) 


Macro Expansion 
:A = 0. 

:A = 0. 

:B = 0. 

:C = 0. 

:C = 0. 

:D = 0. 


Figure 14—Conditional macro expansion examples 


expansion. When the COND statement expression evaluates 
to false, only those statements between the ELSE state¬ 
ment, if included, and the ENDC statement are included in 
the macro expansion. 

String terms in COND statement expressions are either 
quoted strings or formal arguments of the host macro defi¬ 
nition which are replaced at macro expansion time by 
strings. 

Two special forms of numeric terms may appear in COND 
statement expressions. #NARG is the number of actual 
arguments supplied in the current call on the host macro. 
The numeric value of #VAL(arg) is determined as follows. 
Usually, arg is a formal argument in the host macro defini¬ 
tion. After argument substitution, if arg is a numeric con¬ 
stant, then that is the numeric value of #VAL(arg); other¬ 
wise the macro expansion terminates in error. Figure 14 
illustrates conditional macro expansions. 


EXAMPLES 


Macros 

The primary role of macro definitions (Figure 12) is that 
of an efficient shorthand notation for describing test cases. 
Macro definitions are entered in a macro definition library 
under the symbolic name specified on the macro statement. 
When invoked by a reference to the macro name, formal 
arguments appearing in the text of the macro definition are 
replaced by actual arguments supplied by the call. The mod¬ 
ified text of the macro definition then replaces the macro 
call statement. 

The conditional construct (Figure 13) allows statements 
to be conditionally included in a macro expansion. The Boo¬ 
lean expression in the COND statement is evaluated each 
time its host macro is expanded. If the expression evaluates 
to true, all statements following the COND statement, down 
to the ELSE statement, if present, are included in the macro 


The test procedure example in the beginning of this paper 
illustrates the basic structure of test case definitions, the use 
of macro definitions for replicating test cases, I/O simulators 
and subprogram stubs. Two additional examples in this sec¬ 
tion illustrate common styles used for writing test proce¬ 
dures. 

Figures 15 and 16 illustrate the use of I/O simulators for 
testing I/O-oriented target modules. The target module of 


1 READ (12,100,END=300) X,Y 
200 • 

300 WRITE (6,150) W 
STOP 
END 

Figure 15—An I/O-oriented target module 
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MODULES MAIN 

c 

C INPUT VALUES 
C 

I/O SIMULATOR 12 NO REWIND 
RECORD (1)# 1,0.0,0.0 
RECORD (2) #END 
RECORD (3) #1, 0.01,0.00001 
RECORD (4) #1, 0.01,0.00002 
RECORD (5) #END 
RECORD (6) #1, 1.0,5.0 
RECORD (7) #, 1, 2.0,4.0 
RECORD (8) #1, 3.0,3.0 
RECORD (9) #END 
RECORD (10) #1, 10000.0,0.00001 
RECORD (11) #END 

1 FORMAT(‘BASE’,F12.6,10X,‘VALUE’,F12.6) 

END 

C 

C OUTPUT VALUES 
C 

I/O SIMULATOR 6 NO REWIND 
DIMENSION X(4) 

DATA X/0.0, .0.0132,2.5,1.0/ 

RETRIEVE (#1) VAL 

IF (ABS(VAL-X(#TEST)) .GT. 0.00005) ABORT 
RETURN 

1 FORMAT(‘VALUE =’,F14.8) 

END 

C 

C DEFINE GENERAL TEST FOR MAIN 
C 

MACRO TSTPRG(ZP) 

VERIFY AT :200 (:ABS(Z-ZP) .LT. 0.0005) 

EXECUTE 

MEND 

C 

C FOUR TEST CASES 
C 

TSTPRG(1.0) 

TSTPRGO.001) 

TSTPRG(5.0) 

TSTPRGO00.0) 

FIN 

Figure 16—A test procedure for the target module of Figure 15 


. Figure 15 is a main program that reads formatted records 
from logical unit 12 until an end-of-file condition is encoun¬ 
tered. After some processing, the program writes one for¬ 
matted record on logical unit 6 and terminates. 

The test procedure in Figure 16 exercises the single target 
module, MAIN. Four test cases each supply inputs to MAIN 
on logical unit 12 and verify its outputs on logical unit 6. 
Additionally, a non-terminal assertion is verified each time 
control reaches MAIN:200. 

Figures 17 and 18 contains another example illustrating 


SUBROUTINE SUB2(M) 
DIMENSION M(8,3) 


Figure 17—A target module 


MODULES SUB2 
C 

C INPUT VALUES 
C 

I/O SIMULATOR 10 NO REWIND 
RECORD (1:3) 0,0,0,0,0,0,0,0 
RECORD (4:6) 1,1,1,1,1,1,1,1 
RECORD (7:9) 1,0,0,0,0,0,0,1 
RECORD (10:12) 0,1,1,1,1,1,1,0 
END 
C 

C OUTPUT VALUES 
C 

I/O SIMULATOR 20 NO REWIND 
RECORD (1) 1,0,4,8,0,1,5,0 
RECORD (2) 1,1,0,1,5,3,6,0 
RECORD (3) 2,0,1,1,8,1,6,4 
RECORD (4) 7,9,1,6,4,6,6,9 
RECORD (5) 1,7,9,1,4,1,9,4 
RECORD (6) 6,2,5,9,0,3,6,2 
RECORD (7) 0,7,2,0,9,6,9,9 
RECORD (8) 9,5,7,0,9,1,2,9 
RECORD (9) 1,9,0,7,0,0,2,2 
RECORD (10) 3,6,8,4,6,5,7,3 
RECORD (11) 2,5,5,9,5,8,5,3 
RECORD (12) 9,3,3,0,9,9,5,8 
END 
C 

C VERIFY OUTPUTS 
C 

FUNCTION OUTVFY 

WRITE (#20) ((:M(I,J),J= 1,8),1=1,3) 

RETURN 

END 

C 

C DEFINE GENERAL TEST FOR SUB2 
C 

MACRO TSTSB2 

READ (#10) ((:M(I,J),J=1,8),1=1,3) 

VERIFY (OUTVFY) 

EXECUTE 

MEND 

C 

C FOUR TEST CASES 
C 

TSTSB2*4 

FIN 

Figure 18—A test procedure for the target module of Figure 17 


the use of internal references to I/O simulators for generating 
large amounts of data for test case initialization and verifi¬ 
cation. The target module, SUB2, of Figure 17 performs a 
transformation on the array M. 

The test procedure of Figure 18 contains four test cases, 
each of which supplies SUB2 with 24 input values for the 
array M and verifies the contents of M returned by SUB2. 
The input and output values are contained in two I/O sim¬ 
ulators read only by the test procedure. 

DISCUSSION 

The notation that software tests can be specified in a 
formal language, just as are computer programs, will likely 
improve the quality of software test design since, for the 
first time, software tests reside in a medium (the test pro- 


END 
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cedure) that is both executable and readable and therefore 
available for peer review and criticism. Automatic software 
test drivers should also influence the quality of software test 
design by providing feedback on the degree of testing 
thoroughness actually achieved by a test procedure. 

Perhaps the strongest effect of automatic software test 
drivers and formal test procedures will be on the mainte¬ 
nance of production software. It is widely agreed that the 
majority of errors appearing in production software are in¬ 
troduced as unintended side effects of post-release program 
modifications. This is due, in large part, to the absence of 
a convenient mechanism for preserving test cases through¬ 
out a program’s life-cycle so that they can be used to check 
out post-release program modifications. In order to effec¬ 
tively retain software tests for post-release regression test¬ 
ing, it is necessary that someone other than the original test 
designer be able to execute them and verify the correctness 
of test results. Formal test procedures should facilitate the 
retention of software test cases throughout the entire life 
cycle of computer programs since they are completely self- 


contained, they execute automatically without special test 
set-up instructions, and they are self-checking. The adoption 
of automatic software test drivers should therefore lead to 
significant software cost savings since it is often estimated 
that up to 70 percent of the life-cycle costs of computer 
software are attributable to testing and debugging. 
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As the number of institutions of postsecondary education 
facing financial exigency increase, a method of tracking the 
progress of software projects in a systematic and unbiased 
manner is extremely critical. The software quality plan com¬ 
monly found in business and industry can aid the college 
and university in developing guidelines and tests for the 
software development process, thus reducing costs. 

Traditionally, testing and evaluation of software has been 
a minor or non-existent part of the programming effort 
within a university. If testing and evaluating does exist, it 
has normally been carried out by the programmer respon¬ 
sible for project development or by his close associate. This 
could mean that objectivity and selectivity in testing and 
evaluating can be lost because of a programmer’s proximity 
to his work. This could also mean that more of the operating 
budget of the computing center has to be spent on testing 
and debugging after installation has been carried out. Testing 
and debugging in the user’s environment can cause unnec¬ 
essary problems in public relations and system acceptance. 
Some industrial companies, such as TRW, Inc. and the NCR 
Corporation, have recognized this problem and have estab¬ 
lished formal Software Quality Assurance (SQA) depart¬ 
ments or interest groups whose responsibilities include the 
task of assuring that software is free from as many errors as 
possible. In carrying out their functions, the SQA organi¬ 
zation utilizes a formal plan or guideline known as the Soft¬ 
ware Quality Plan (SQP). The SQP is a document prepared 
by each of the participating internal departments within 
computer services (programming, operation, training, pro¬ 
duction, etc.) and the customer or client. The plan, once 
prepared, spans the entire development cycle and becomes 
a coordinating agent between all interested groups in the 
establishment of activities, schedules, commitments and 
budgets. 

Within the institution of postsecondary education, re¬ 
sponsibility for the development of the SQA should be under 
the direct supervision of the Director of Software Quality 
Assurance or his designate. In institutions where there is no 
formal quality assurance office, the plan could be produced 


by the internal auditor or at least by the chief officer in 
charge of computing services. Although the SQP is initiated 
early in the development cycle, it must be updated as the 
project moves from one development phase to another. 
When writing a SQP, the following criteria must be consid¬ 
ered: financial and budgetary analysis, manpower require¬ 
ments (by departments), program dependencies (hardware 
and software), milestone schedules, and commitments by 
the intended user in relation to their support and acceptance 
criteria. Test plans and responsibilities should clearly be 
outlined including the control procedure, methodology, lim¬ 
itations, and acceptance criteria for each test to be used. It 
is suggested that there be at least a review or test during the 
feasibility stage, in order to check architectual proposals; 
the coding stage, in order to assure proper design; pre-in¬ 
stallation, in order to exercise programs with near real data; 
installation to insure users are adequately trained and that 
proper documentation is available. Resource requirements, 
in the sense of stating needed hardware, software, special 
test tools, additional manpower and required documentation 
should also be provided in the SQP. 

In essence, software quality within the university frame¬ 
work is a necessity, especially in light of financial exigency. 
This quality can be attained by better control during the 
project development process and by assuring test objectivity 
by having the evaluation of software carried out by non¬ 
partisan people—people not involved with the actual writing 
and coding of the software. These non-partisan persons 
would be part of a Software Quality Department and would 
be responsible for the implementation of a Software Quality 
Plan, that is, a guideline for the monitoring of software 
quality throughout the development process. Above all else 
the plan would describe the various tests to be conducted, 
responsibilities in carrying out those tests, and the results 
expected. This plan is not intended as a panacea for all 
postsecondary quality ills, but is intended as a starting point 
for the assuring of quality software in the university envi¬ 
ronment by reducing costs normally incurred when errors 
are corrected on the “fly.” 
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INTRODUCTION 

When testing software the major question which must al¬ 
ways be addressed is “If a program is correct for a finite 
number of test cases, can we assume it is correct in gen¬ 
eral.” Test data which possess this property is called Ade¬ 
quate test data, and, although adequate test data cannot in 
general be derived algorithmically, 1 several methods have 
recently emerged which allow one to gain confidence in 
one’s test data’s adequacy. 

Program mutation is a radically new approach to deter¬ 
mining test data adequacy which holds promise of being a 
major breakthrough in the field of software testing. The 
concepts and philosophy of program mutation have been 
given elsewhere, 2 the following will merely present a brief 
introduction to the ideas underlying the system. 

Unlike previous work, program mutation assumes that 
competent programmers will produce programs which, if 
they are not correct, are “almost” correct. That is, if a 
program is not correct it is a “mutant”—it differs from a 
correct program by simple errors. Assuming this natural 
premise, a program P which is correct on test data T is 
subjected to a series of mutant operators to produce mutant 
programs which differ from P in very simple ways. The 
mutants are then executed on T. If all mutants give incorrect 
results then it is very likely that P is correct (i.e., T is 
adequate). On the other hand, if some mutants are correct 
on T then either: (1) the mutants are equivalent to P, or (2) 
the test data T is inadequate. In the latter case, T must be 
augmented by examining the non-equivalent mutants which 
are correct on T: a procedure which forces close examina¬ 
tion of P with respect to the mutants. 

At first glance it would appear that if T is determined 
adequate by mutation analysis, then P might still contain 
some complex errors which are not explicitly mutants of P. 


* This research was supported in part by NSF Grant MCS76-81486 and U.S. 
Army Research Grant DAAG-29-76-G-0338. 


To this end there is a COUPLING EFFECT which states 
that test data on which all simple mutants fail is so sensitive 
that it is highly likely that all complex mutants must also 
fail. 

Readers wishing a further exposition of the ideas of mu¬ 
tation and substantiation of the assumptions made are re¬ 
ferred to References 2 and 10. 


THE SYSTEM 

A pilot system has been built to implement mutation anal¬ 
ysis on programs written in a subset of FORTRAN. The key 
features of this system are summarized in Figure 1. The 
system itself consists of 10,000 lines of FORTRAN code, 
and required six man months to design, implement and 
debug. 

Notice we claim the system is man/machine interactive. 
In general an attempt is made to assign tasks to both the 
user and the machine processors which are best suited to 
using their particular capabilities. One way to see this is to 
view the system as a sort of “Devils Advocate”, which 
when confronted with a program asks very difficult ques¬ 
tions about the motivation behind it (“why did you use this 
type of statement here, when an alternative statement works 
just as well?”). The job of the human is then to provide 
justification (in the form of test data), which will give an 
answer to such questions. 

An overview of the structure of the system is given in 
Figure 2. We point out that the language FORTRAN was 
chosen for the first implementation merely as a matter of 
convenience since it is in common use and there is a large 
body of software in existence to experiment on. The heart 
of the system (roughly that shown within the dotted box) is 
however, language independent, and given a sufficiently 
general internal form to implement a new language one 
would merely write a new input/output interface. Projects 
are currently under way to implement mutation analysis on 
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INTERACTIVE 
MACHINE INDEPENDENT 
LANGUAGE INDEPENDENT STRUCTURE 
MODULAR DESIGN 

INTENSIVE MAN/MACHINE INTERACTION 
Figure 1—Key features of the pilot mutation system 


COBOL and C (an ALGOL like language) using the struc¬ 
ture represented by the box contained in the dotted lines. 

An attempt was also made to keep the structure of the 
system largely machine independent. The system was orig¬ 
inally programmed to run on a PDP-10 at Yale University. 
Currently we are in the process of transferring it to a CDC 
7600 at the Georgia Institute of Technology. 

A single run of a mutation system divides naturally into 
three phases the RUN PREPARATION phase, in which the 
necessary variables to send to the mutation executor are 
defined, the MUTATION phase, in which the actual muta¬ 
tions are produced and executed, and the POST RUN phase, 
in which results are analyzed and reports are generated. In 
the following we will describe in more detail the structure 
and effects of each phase. 

The role of the run preparation phase is to initialize the 
various files and data buffer areas used by the mutation 
executor. It is characterized by a very interactive nature. 
The first object the user is requested to supply is the name 
of the file on which the FORTRAN subroutine resides. Then 
depending on whether PIMS has been run previously on this 
routine (in which case the internal form is stored on one of 
the many files PIMS constructs, see below) the subroutine 
is parsed into a concise internal format which is subse¬ 
quently interpreted to simulate execution of the program. A 



Figure 2 


IF (A .LT. X(2)) P = 1 

[SCALAR. A] 

[ARRAY1. X] 

[CONSTANT.2] 

[AOP.SUBSCRIPT] 

[ROP.LT] 

[TRF.O] 

[SCALAR, p] 

[CONSTANT.1] 

[ASSIGN.0] 

Figure 3 

fragment of the internal code generated for a given statement 
is shown in Figure 3. 

The user is then interactively prompted for the test data 
on which the program and mutants are to be tested. After 
each test case has been specified the original program is 
executed on the test case and the results displayed so that 
the user may satisfy himself that the results produced are 
indeed correct. 

After the test data has been entered the user is prompted 
for a listing of which mutant operators he wishes to enable. 
At present there are 25 mutant operators. These range from 
very simple low level ones, such as replacing each data 
occurrence (where a data occurrence is a scalar, constant 
or array reference) with all other syntactically correct data 
occurrences, to very high level mutations, such as deleting 
statements or altering the control structure of the program. 
A more detailed description of the mutations performed can 
be found in Reference 3. 

Instead of constructing multiple copies of the program, 
for each mutant a short (four word) description of the mu¬ 
tation to be performed is kept. Each time the mutant is to 
be run the original program is then mutated according to the 
contents of this descriptor. 

After the user has specified to the system his program, 
test data and the mutant operators he wishes applied, the 
system then enters the MUTATION phase. During this 
phase there is no user interaction. Mutation descriptor rec¬ 
ords are read in, one by one, and the mutation is produced. 
The mutant program is then executed on the test data and 
marked either “dead,” meaning it produced results differing 
from the original program on at least one test case, or “liv¬ 
ing.” A dynamic record is kept of the number and percent¬ 
age of living mutants of each mutation type. 

When all the mutant programs have been tested the post 
run phase is entered. In this phase statistics are displayed 
indicating the results of the mutation run. In addition the 
user can interactively view descriptions of those mutations 
which have survived. He can also specify that certain re- 
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RUN 

PREPARATION 

PHASE 


MUTATION POST RUN 

PHASE PHASE 



Figure 4 


Display results 
and 

Produce permanent 
records 


ports be generated in order to provide a detailed permanent 
record of the mutation run. 

At this point, or at a later date, the user can re-run the 
system and augment his test data in an attempt to make the 
remaining mutants fail. He may also specify that additional 
mutant operators be applied to the program. This cycle can 
continue until the user is satisfied that the current test data 
adequately tests his program. 

There are several files the system produces in order to 
store information from one run to the next. These are shown 
in Figure 4, which outlines the major functions of each 
phase. The internal form file stores the parsed version of the 
program. The test data file stores for each test case the test 
data input and the results of execution of that test data. The 
mutants information file keeps the mutant descriptor records 
plus various other counts on what types of mutants have 
been produced. 

A COMPARISON OF PIMS TO OTHER DATA 

TESTING SYSTEMS 

Various systems have been discussed in the literature for 
increasing confidence in the adequacy of test data, as the 
PIMS system does, or automatically constructing test data 


which meets some criterion. In this section we will report 
on experiments which show that the PIMS system is an 
improvement in this area over other systems which have 
been proposed. 

The most widely known method of constructing test data 
automatically are those systems which utilize path analy¬ 
sis. 4-7 Essentially, these procedures attempt to construct 
data which force each statement to be executed at least 
once, and furthermore which transverse each feasible flow 
path through the code at least once. In some cases, such as 
loops, only an approximation to this can be made as the 
number of flow paths may be infinite. Here it is usual to just 
construct data which cause the loop to be executed at least 
twice. 

These same objectives are met with mutant analysis in a 
number of ways, some directly by mutant operators, others 
indirectly by the coupling effect. There are mutants which 
cause each statement in the original program to be replaced 
by a TRAP statement, a special type of statement which if 
ever executed causes the program to immediately abort. 
Obviously, then if there is some statement in the program 
which is never executed, changing that statement to a TRAP 
statement will not alter the output of the program and hence 
will easily be detected. 

Checking that every decision path is taken is essentially 
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DO 10 1=1.J 


10 CONTINUE 
DO 10 1=1.1 


10 CONTINUE 

A LIVE MUTATION IF THE LOOP IS 
ALWAYS EXECUTED ONLY ONCE. 

Figure 5 

the same as checking that every predicate in the program 
evaluates at least once to both true and false. If this is not 
the case, say the predicate always evaluated to TRUE, then 
we can mutate the predicate in any way we desire as long 
as it retains this property of always remaining TRUE. These 
types of mutations are also usually quite obvious and easily 
detectable. 

Mutation analysis can also insure that each loop is trav¬ 
ersed at least twice. The only way a loop can be traversed 
only once (and all loops must be traversed at least once to 
pass the TRAP statement mutations) is if the terminating 
condition is the same as the starting condition. But in this 
case the mutant which replaces the terminating condition by 
the starting condition will survive (see Figure 5). This is 
once more easily detected. 

With this, mutant analysis possesses all the capabilities of 

SUBROUTINE BSERCH(X.Y.N.A.IHIGH.LOW.ERR) 

INTEGER X(N).Y(N).N.A.IHIGH.LOW.ERR.MID 
C BINARY SEARCH PROCEDURE. IF X CONTAINS A ON RETURN 

C X(IHIGH) = A.IHIGH=L0W. IF NOT X(LOW) < A < X(IHIGH). 

C IF A IS OUT OF RANGE ON RETURN ERR CONTAINS 1 

ERR = 0 

IF ((X(l)-A).GT.O) GOTO 11 
IF ((A-X(N).LE.0) GOTO 5 

11 ERR = 1 
RETURN 

5 LOW = 1 
IHIGH = N 

6 IF ((IHIGH-LOW-1).NE.0) GOTO 7 
RETURN 

7 MID = (LOW+IHIGH)/2 

IF ((A-X(MID)).GT.0) GOTO 10 
IHIGH = MID 
GOTO 6 

10 LOW = MID 
GOTO 6 
END 

Figure 6 


Replace IF ((IHIGH-LOW-1).NE.O) GOTO 7 
by IF ((IHIGH-LOW-1).GT.O) GOTO 7 

Replace MID = (LOW+IHIGH)/2 
by MID = (LOW+IHIGH)-2 

Figure 7 

path analysis systems which have been discussed in the 
literature. 

Another class of systems for which extensive claims have 
been made are those which detect uninitialized variables and 
dead code. 8 Uninitialized variables are caught as a conse¬ 
quence of the interpretation process in the mutant system. 
Dead code is easily caught since an assignment made to a 
dead variable can be mutated in any way whatsoever and 
the program will remain the same. 

A third class of systems for which there has recently been 
much discussion involves symbolic execution of the program. 
In one study 9 Howden analyzed 12 programs containing a 
total of 22 errors. He found that symbolic execution would 
catch 13 of those errors, while path analysis would discover 
only nine. In a similar study we estimated that mutation 
analysis, using only the mutant operators in the present 
PIMS system, would uncover 18 of the 22 errors. Of the 
remaining four, three would probably be discovered if we 
added two new mutant operators which the authors simply 
had not thought about. Hence, mutation analysis is in certain 
cases an improvement over symbolic execution. 

As an example of the very subtle errors which mutation 
analysis can discover consider the program to perform bi¬ 
nary search shown in Figure 6. If it happens that N= 1 when 
the subroutine is called (i.e., the vector to be searched 
contains only a single element) then it is not difficult to see 
that the program will loop indefinitely. It is not clear that 
either symbolic execution or path testing would be sufficient 
to discover this error. 

When mutant analysis is applied to this program there are 
two mutants generated (shown in Figure 7) which can only 
be eliminated by a test case consisting of one element. 
Hence the error is easily detected using mutant analysis. 
(There is a second error in this program which is also un¬ 
covered by mutant analysis. The discovery of that second 
error is left to the reader). 

FUTURE WORK 

There are several directions in which work is currently 
being pursued with respect to mutation analysis and the pilot 
mutation system. The most obvious is to show how a similar 
system might be built around another language, and research 
is under way to construct systems for COBOL and for C. 

Another area of study is the design of an easy to use 
language for the description of test cases which allows for 
a variety of features. Test datasets can often be quite 
lengthy, yet two test cases can be very similar. Also, a user 
often wishes just to construct a number of random test cases 
following some specification. (Some of the pitfalls of using 
random data to test programs are discussed in Reference 10 
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where it is seen that mutation analysis can help in deriving 
“good” random test data.) Finding an easy yet powerful 
method of solving this problem is the goal of one area of 
research. 

Finding a method to detect equivalent mutants is another 
area currently being pursued. It is often the case that a 
mutation will not produce a significantly different program 
(replacing the sequence 1= 1 J= 1 with the sequence 1= 1 J=I 
is a trivial example). We have observed that programs tested 
have between one and two percent equivalent mutants. A 
method to automatically detect and remove equivalent mu¬ 
tants would allow us to provide even more significant meas¬ 
ures of the adequacy of a test data set. 

We point out that as a consequence of the modular design 
of the pilot system either of the above two major extensions 
can be added without a significant reprogramming effort. 

A final area of current interest is the study of mutant 
operators. Certain operators seem to have a much greater 
ability to detect errors then others. Analysis of data along 
these lines would allow us to discover an order of application 
of mutant operators which would maximize the cost/benefit 
ratio. 

CONCLUSIONS 

It has been shown that the ideas of program mutation can 
be quickly and easily implemented as an interactive system 
for program testing. The resulting system represents a cost 
effective engineering approach to testing real world soft¬ 


ware. Large subroutines (over a hundred statements long) 
have been analyzed by our system with relative ease. 

Mutation is a method of program testing which will sig¬ 
nificantly raise the level of reliability in both new and exist¬ 
ing software, and is a major advance in the area of software 
testing. 
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PANEL OVERVIEW—Victor R. Basili 

The development of correct, reliable, less expensive soft¬ 
ware continues to be a major problem. A great deal has been 
written and said about various techniques and methodolo¬ 
gies for software development and how they are meant to 
aid in the development process. Unfortunately, most of the 
available material is by the author of the technique, be it 
individual or company. This does not always allow the out¬ 
side user of the technique a fair appraisal or full understand¬ 
ing of how good the technique is, how to use it, and how to 
adjust it to his environment. This is true for several reasons. 
First, the author’s experience is often limited to a specific 
application or set of applications and specific environments. 
There are some genuine questions that arise when taking a 
technique and moving it to a new application or environment 
that the developer of the technique had not anticipated. 
Second, the author does not always tell the prospective user 
everything he needs to know. Often this is just due to a lack 
of documentation, or a set of basic assumptions and back¬ 
ground that the author did not realize was even necessary. 
Lastly, one cannot normally expect the author to emphasize 
the weak points or problems in the methodology. That is 
just human nature. 

The purpose of this panel and the' following set of papers 
is to discuss a set of techniques available in the open liter¬ 
ature, some very new, some that have been around for 
awhile, and ask for an analysis and evaluation by current 
users. Each of the panelists is not the author of the meth¬ 
odology but a member of a company that is using the meth- 
dology or overseeing a contract on the use of the method¬ 
ology. Some of the users are old hands at the technique; 
some are novices. I have asked each of the panelists to 
prepare a short paper covering a brief description of the 
technique and an evaluation of the technique in a real en¬ 


vironment. Suggested ideas to be included in the paper and 
the oral presentation are given below. 

I. THE TECHNIQUE 

The techniques you have been using. 

A description of the project you are using it on. 

The phase of the project in which it is used. 

The phases of the project it affects. 

An overview of the technique. 

Why you chose it. 

The extent to which you are using it. 

II. EVALUATION OF THE TECHNIQUE IN A 
REAL ENVIRONMENT 

What have you felt are its good points and how 
they have shown to be good. 

What have you felt are the weak points and why. 

How you have adapted it to your environment. 

Would you use it again and if so how would you 
change or have you already changed the way it 
should be applied. 

What would you recommend to someone else ap¬ 
plying it. 

Certainly lots of techniques could have been covered but 
there was limited time and space available. The techniques 
chosen were based partially on my own interest and partially 
on the availability of people willing to discuss specific tech¬ 
niques. Discussants and topics include: 

PSL/PSA—Donald J. Reifer, 

SADT—Donn Combelic, ITT Telecommunications 
Structured Design—Dr. J. A. Rader, Hughes Aircraft 
Jackson’s Methodology—Clifford M. Bernstein, EXXON 
Corporation 

Boeing’s IPAD Methodology—Susan Voigt, NASA Lan¬ 
gley Research Center 

Correct Program Design—F. Terry Baker, IBM Corpo¬ 
ration 

The methodologies deal with various phases of the soft¬ 
ware development process, from requirements to program 
development, some emphasizing one specific phase and 
some covering several phases. PSL/PSA and SADT deal 
predominantly with the requirements phase. Structured De- 
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sign, Jackson’s Methodology, and Correct Program Design 
deal with various aspects of design and development. The 
Boeing IPAD Methodology covers the gamut from require¬ 
ments to completed product. 

What follows is a position paper for each of the panelists. 
Each is meant to be self contained, complete with references 
to the appropriate material. 


EXPERIENCE WITH PSL/PSA—Donald J. Reifer 

INTRODUCTION 

This paper briefly describes the experience we have had in 
using the University of Michigan developed Problem State¬ 
ment Language (PSL)/Problem Statement Analyzer (PSA). 1 
We are using these computer assisted requirements tools to 
document and analyze operational flight software require¬ 
ments developed for the Titan 34D segment of the Space 
Transportation System. The Titan 34D segment is providing 
real-time guidance, checkout and control requirements for 
implementation on the Interim Upper Stage. These boost 
phase requirements are highly time-critical and computation- 
bound. In addition, they must be documented in accordance 
to the Software Part I Specification format of MIL-STD- 
483. 2 

PSL/PSA DESCRIPTION 

PSL is a machine processable language for expressing 
functional and performance requirements for a data system 
in a rigorous and uniform manner. PSL contains a set of 
simple declarative statements that allow you to name con¬ 
ceptual objects in a proposed system, describe properties of 
these objects and display relationships between them. 3 PSA 
is a software package that accepts PSL statements, analyzes 
each statically for correct syntax, generates a requirements 
data base from the input statements, performs consistency 
and completeness analyses on the data base and generates 
various kinds of reports and the requirements document. 
The command language with which users invoke the PSA 
services is also simple and user oriented. 4 

SELECTION CRITERIA 

We selected PSL/PSA from several alternatives for sev¬ 
eral reasons. First, it is commercially available and sup¬ 
ported. There is a commitment to keep the system current 
and fix errors. Second, it is well documented and good 
training is available. Third, it is operational on many large 
computing environments including the IBM 370, CDC 6000/ 
7000 and Honeywell series. Lastly, it is currently being used 
by several diverse commercial and military users. The multi¬ 
user feedback has made the system mature faster than the 
alternatives. Readers are directed to Davis and Vick’s paper 5 


for an excellent comparison of PSL/PSA with other tech¬ 
niques. 

We elected to use the tool on the Titan 34D application 
because we felt there was relative low risk inherent in their 
requirements. Titan 34D requirements are based on flight- 
proven equations and logic. We felt that the potential ben¬ 
efits justified the risk associated with using a new tool to 
document known and mature requirements. 

EXPERIENCE 

The ISDOS staff of the University of Michigan was re¬ 
tained by the Martin Marietta Corporation to install the PSL/ 
PSA system on their CDC 6500 computer and conduct train¬ 
ing classes. Installation began in April 1977. Several prob¬ 
lems occurred as the package was adapted to the machine. 
First, the number of pages that resided in core had to be 
adjusted in order to make the system more efficient in terms 
of internal charging algorithms. Second, a machine-depend¬ 
ent executive routine that generated control instructions to 
do routine calling had to be developed. Training was held in 
the April/May timeframe with general orientation classes, 
user classes and maintenance classes being held. 

The PSL/PSA system was then used to generate opera¬ 
tional requirements for the Titan 34D segment of the Space 
Transportation System. The positive results of its use can 
be summarized as follows: 

1. It forced the user to understand his problem and ad¬ 
dress it in a disciplined manner. 

2. It documented requirements in a uniform manner and 
eased the task of document maintenance. 

3. It assisted in the identification of errors primarily in 
the areas of incompleteness and inconsistencies. 

The negative features of the system are as follows: 

1. The system was too general to be used for a specific 
application. An internal groundrules document had to 
be developed to tell the user what to do and what not 
to do (e.g., limited number of attributes, naming con¬ 
ventions for decimal numbers, keyword conventions, 
etc.) 

2. The PSL/PSA systems used large amounts of computer 
time that were not planned for. 

3. The system is oriented towards a business environment 
and, makes assumptions that make it difficult to de¬ 
scribe requirements for realtime systems (e.g., concept 
of counters and interfaces hard to describe, notation is 
not compatible with scientific symbols, etc.). 

4. The system was hard to sell to the users who were 
engineers. Therefore, changes had to be made to over¬ 
come user criticism (e.g., illegal characters like / in ft/ 
sec had to be made legal). 

5. The system had to be extended in order to provide I/O 
tables required by the B5 specification 7 format. A 
FORTRAN program was developed to search existing 
files and generate the tables. 
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The positive benefits resulting from use of PSL/PSA com¬ 
pensated for the negative experiences. Plans have been 
made to utilize the system to describe software requirements 
for other military systems. 
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EXPERIENCE WITH SADT—Donn Combelic 


BACKGROUND 

SADT, Structured Analysis and Design Technique, is a 
registered trademark of SofTech Inc., Waltham, Mass. ITT 
has used the “Structured Analysis” part of SoftTech’s 
SADT since early 1974. In mid-1975 we began to develop 
for real-time switching software our own structured design 
technique, called FP2 for Functions-Processes-Flowgrams- 
Programs, based upon precepts and syntax of Structured 
Analysis. Thus the technique we are now using for analysis 
and design is called SA/FP2. 


GENERALITIES 

The principal basic ideas of the technique are: determine 
the “what” before the “how”, decomposition from the top 
down to reveal successive levels of detail, output in the form 
of diagrams each of which gives a limited amount of detail, 
each diagram is critiqued in writing by one or more persons 
other than the author of the diagram, needed information 


unknown to an author is obtained by interviews with outside 
experts. Each diagram is comprised of boxes that represent 
“activities” interconnected by arrows that represent “data” 
used by an activity for input, output and control. A box plus 
its arrows constitutes the “context” of that activity—it is 
that (bounded) context which is decomposed to understand 
and show more detail in a diagram at the next level. 


OVERVIEW OF SA/FP2. 

SA/FP2 is carried out in five phases, one of which is 
concurrent with two of the others. The first phase is that of 
Structured Analysis (SA); the remainder are those of design, 
that is, FP2. A brief summary of each of the five phases is 
given in the following paragraphs. 

Structured analysis phase 

SA is ideally applied to the total system. However, in 
most applications we have applied it only to real-time 
switching software. In such a case, the primary inputs are 
a list of customer requirements plus functional specifications 
of the telephone hardware of the system. The output is a 
Functional Requirements Model (FRM) in the form of a set 
of activity diagrams many of which are accompanied by a 
page or two of explanatory text and definitions of terms. 
The FRM shows what functions the software must contrib¬ 
ute in addition to those of the telephone hardware in order 
to fulfill the customer functional requirements. To the great¬ 
est extent possible, software design considerations, such as 
data layouts, scheduling, priorities, handling of queues, buff¬ 
ers and computer peripherals, are kept out of the FRM. 
Thus the FRM emphasizes the “what,” not the “how”. 

Transfer phase 

This is the first phase of design. Its principal inputs are 
the FRM and the software design constraints. Typical con¬ 
straints are: choice and arrangement of computers, com¬ 
puter peripherals, programming language, requirements for 
traffic handling, engineerability, extensions, maintainability, 
etc. The outputs of the transfer phase are a high level data 
layout model and a single level “action group” model. (A 
software action is defined as a sequence of instructions 
which, once started, can run to completion without waiting 
because all needed inputs were available at its start.) For 
each action group, a convenient set of one or more contig¬ 
uous activities, along with the data arrows at the boundaries, 
is selected at an appropriate level of detail from the FRM 
and transferred (as a single box) to the action group model. 
The action groups are interconnected as the corresponding 
sets of activities were interconnected in the FRM. The high 
level data layout model is developed before and during the 
transfer procedure. The transfer phase is complete when all 
activities of the FRM have been accounted for in the action 
group model. 
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Action group decomposition phase 

All action groups are decomposed to the level of individual 
actions. The output is a set of activity diagrams where each 
box at the lowest level of detail represents an individual 
action. An additional output is further detail of the data 
layout. During this and the preceding phase the decompo¬ 
sition rules are the same as for the SA phase, but the SA 
syntax is augmented to handle action starts and completions. 

Flowgram phase 

Each action is decomposed, according to its implicit con¬ 
trol flow sequence, down to the level of individual routines, 
each of which appears as a separate box on the lowest level 
diagrams. The control flow is shown on each diagram in a 
special syntax, hence the name “flowgram.” The output of 
this phase is a “flowgram model” for each action. The 
previous syntax is augmented to handle control flow. It turns 
out that when the control flow sequence and the individual 
routines are coded the resulting set is a structured program. 
Thus there is a structured program for each action. 

Coordinator phase 

The coordinator is that software which, among other 
things, starts all actions and to which all actions return upon 
completion. It thus includes the functions of scheduling, 
handling of queues and management of memory. It is con¬ 
venient to include within the coordinator the treatment of 
interrupts and the handling of telephone and computer pe¬ 
ripherals. It is interesting to note that none of these functions 
relate directly to the customer’s basic functional require¬ 
ments, rather they all depend on the nature of the system. 
The functional requirements for the coordinator begin to 
emerge as early as the SA phase, become more clear by the 
end of the transfer phase, but cannot be known completely 
until the action group decomposition phase is finished. By 
that time the coordinator can be completely specified and 
designed. Note that the techniques described for the pre¬ 
ceding phases can be applied to the analysis and design of 
the coordinator. 


EARLY EXPERIENCE—1974-1976 

Development of the FP2 design methodology reached the 
point where it could be used in practice only at the beginning 
of 1977. Thus all our prior experience was limited to Struc¬ 
tured Analysis as taught by SofTech and refined by ITT and 
SofTech together. We adopted SA in early 1974 for two 
main reasons. First, it provided a disciplined way of under¬ 
standing requirements in detail before starting design. Sec¬ 
ond, it offered a method which promoted teamwork. The 
latter was a particularly difficult problem on some of our 
projects in Europe where a team would consists of members 
with widely varying experience from up to eight different 


ITT companies speaking six different languages. Of the ap¬ 
proximately twenty ITT projects where SA has been used, 
all but one are in Europe. 


Strong and weak points 

A partial list, derived from our early experience with SA, 
is as follows: 

• SA estimated to decrease overall software development 
cost by at least 20 percent and significantly improve 
software quality—estimated 2 to 10 times less bugs 
found during integration testing, varies with project. 

• Very definitely promoted teamwork. 

• Hard to think all the time in purely functional terms. 

• The written comments (by other than the author) re¬ 
quired for each diagram resulted in continual review, in 
effect “walkthroughs.” (Note : Commentators should 
normally be other authors in the same team.) 

• Interviews of outside experts proved efficient method 
of obtaining specialized information. 

• Forced making high level decisions early, thus provid¬ 
ing a sound basis for later lower level decisions. 

• Encouraged agreement on requirements before start of 
design. 

• Lack of follow-through on design methodology (later 
overcome by FP2) was bothersome. 

• Permitted non-software people to understand the con¬ 
tributions of software functions to system operation. 

• Much more useful information per sheet (diagram) than 
with documents in prose. 

• Provided easy way of measuring progress during anal¬ 
ysis. 

• SA excellent for many applications other than real-time 
switching software, but space does not permit elabo¬ 
ration. 


Some mistakes made and lessons learned 

A partial list follows: 

• Method was oversold in the beginning as a panacea. 

• Proper use of SA requires a fundamental change in 
mental outlook—difficulties of achieving such a change 
were underestimated. 

• Mere “training” is inadequate—“education” is re¬ 
quired. 

• Potential authors must be selected on the basis of in¬ 
telligence and willingness to try a new way rather than 
purely on experience. 

• Constructive critics of the method are to be cherished, 
but destructive critics must be eliminated from the SA 
group as soon as they surface. 

• Method takes much more time before design starts than 
previous methods. Inevitable if requirements to be thor- 
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oughly understood and agreed before design, but causes 
impatience in some participants and management. 

• Need one “friendly” group to try method first. After¬ 
wards it’s better to “offer” the method to other groups 
than to try to “sell” it—hard sell doesn’t work. 

• “Discipline” in use of method is very important: syn¬ 
tax, conventions, rules, author-commentator cycle, 
completion of a level of decomposition before going to 
next, each diagram must increase understanding, etc. 

• Large amounts of paper generated. Project librarian 
must be assigned, as recommended by SofTech. 

• Formal training is lengthy—two to three weeks full 
time—but necessary. 

• Potential authors must have a project in mind during 
training and be assigned full-time to it immediately 
thereafter. 

• Follow on assistance by a trained “monitor” is oblig¬ 
atory during initial application of method. 

• Monitor must confine his attention to proper use of the 
method rather than become involved in the substance 
of the analysis. 

• Training plus monitoring by SofTech is costly: 20 to 40 
thousand dollars per course for up to ten authors, but 
worth it. 

• Ideal team during SA phase for our type of large switch¬ 
ing projects seems to be two or four persons each from 
systems, hardware and software. 


Recent experience—1977 

It is our estimate that about two-thirds of the value of SA/ 
FP2 is in the SA part. Nevertheless, the availability of a 
follow on design method, in our case FP2, makes the appli¬ 
cation of SA easier because the SA authors are more ready 
to defer design considerations to the FP2 phases. 

The development and acceptance of FP2 has proved ex¬ 
tremely difficult and its success has not yet been fully dem¬ 
onstrated. We wanted to use the same basic precepts and 
syntax as in SA so that the designers, some of whom will 
have also participated in the analysis by SA, would not have 
to learn a new syntax and set of principles—we wanted a 
sort of “continuous” method, starting with a list of require¬ 
ments and constraints and ending up with detailed coding 
specifications. This “continuity” and similarity of syntax is 
important to software maintenance personnel and will assist 
comprehension by interested customers. 

This section concentrates on our 1977 experience with 
FP2; subsequent experience will be covered during the pre¬ 
sentation at NCC 78. 

• Underestimated importance of providing detailed guide¬ 
lines for carrying out the Transfer phase. 

• Initially called the output of the Transfer phase, the 
“process model.” “Process” has many meanings and 
caused great confusion. The neutral phrase “action 
group” conveys the intent without confusion. 

• Software design still requires great skill, but FP2 per¬ 


mits easy comprehension after key design decisions 
have been made. 

• FP2 criticized for not rendering high level software de¬ 
sign “semi-automatic” or at least “semi-algorithmic.” 

• Direct coding from flowgrams is in most cases straight¬ 
forward and can be done by programmers other than 
the designers. 

• The fallout of a “structured program” for each action 
has proved very appealing. 

• Test plans can be developed throughout FP2 in increas¬ 
ing detail and related directly to diagrams. 

• Much debugging is done by reference to flowgrams 
instead of listings. 
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EXPERIENCE WITH AN APPLICATION OF 
STRUCTURED DESIGN—J. A. Rader 


INTRODUCTION 

The application of structured design to a 20 man year project 
which generated 100,000 lines of code is described. Included 
are a description of the project, productivity figures, and a 
discussion of strengths and weaknesses of the technique as 
practiced on the project. 


THE APPLICATION 
Introduction 

The Computer Aided Design (CAD) Department in the 
Hughes Aerospace Groups contains about sixty employees. 
The department provides Computer-Aided Design/Test/ 
Manufacturing software and services; it operates and sup¬ 
ports a DEC system 10 computing facility; and it operates 
and maintains a Gerber photoplotter and several digitizers. 

Most of the software is data manipulative in nature—files 
are read; fields are extracted from records and massaged; 
arrays are built, operated on and sorted; reports are gener¬ 
ated; and data bases are accessed and updated. The primary 
language has been and continues to be FORTRAN. In ad¬ 
dition, there is an extensive library of FORTRAN-callable 
assembly language routines to perform bit and character 
manipulations as well as other special functions. Where very 
heavy CPU utilization is expected, assembly language is 
also sometimes employed. 
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Conversion decision 

Several years ago, a corporate decision was made to phase 
out the Honeywell G635, the computer on which the CAD 
System at that time ran. Among the many alternatives con¬ 
sidered for rehosting the CAD System, the one finally cho¬ 
sen was to purchase a DEC system-10 and convert the CAD 
system to run on that computer. The primary reason for 
selecting the DEC-10 was a proven time-sharing capability. 

The conversion from the G635 to the DEC-10 was a con¬ 
version only from the standpoint of function. 

Existing programs were inventoried and for each it was 
decided which would be converted essentially as is, which 
would be modified, and which would be discarded. 


Goals and advanced plan 

As we started to plan, we recognized that it was very 
important to firmly establish our goals, and to determine 
very specific milestones. Thus we would be able to measure 
our progress and to report on it to our management, who 
was picking up the tab. 

Succinctly stated, the major goals were: (1) to create a 
unified system tied together by a central data base; (2) to 
create software that was reliable and maintainable; (3) to 
provide a user interface that was easy to use and that was 
consistent across all software; and (4) to proceed in a manner 
that allowed us to measure how well we were meeting sched¬ 
ules. 

In late 1973, an advanced planning and development ac¬ 
tivity was formed. The advanced planning group was to 
define the overall structure of the system, and to specify the 
procedures to be followed in specifying and implementing 
the system. 

In November of 1973, the author attended a 6-day in-plant 
seminar on structured design. This seminar was taught by 
Larry Constantine and proved to be of immeasurable value. 
The value arose not from revolutionary concepts but rather 
from a well reasoned and coherent discussion of the relevant 
concepts. The ultimate result of attendance was the gener¬ 
ation and documentation of a methodology for practicing 
structured design in our organization. This methodology is 
described in the next section. 


Outputs of advance group 

In the approximately 18 months of its existence, the ad¬ 
vance group produced 4 basic outputs. First, it produced a 
system concept. Second, it defined the standards and pro¬ 
cedures to be followed in rewriting the system. These were 
documented in a Standards and Procedures Manual issued 
to all programmers. Third, it defined most of the applications 
support software. Fourth, it provided a test vehicle for the 
standards and procedures defined, and provided the first 
productivity figures for the methodology. These figures were 
used in estimating the effort required to implement the main 
body of CAD software. 


CAD SYSTEM 
Subsystem (RXXXXX) 
Process (RVWXXX) 


Activity (RVWOOA) 

Module (RVW128) 

Figure 1—Software hierarchy 
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STRUCTURED DESIGN 

The following hierarchy, illustrated in Figure 1, is used to 
identify levels of CAD software: system, subsystem, proc¬ 
ess, activity, and module. System means the whole CAD 
System. A subsystem is a major functional area. Examples 
of subsystems are routing, digital test and simulation, design 
capture, etc. A process performs an identifiable user func¬ 
tion which may be simple or complex. The process level is 
the lowest level visible to the user. 

A process consists of one or more activities, where an 
activity is an executable core image or job step. A module, 
which is the lowest level, is a single FORTRAN or assembly 
language subroutine. Our FORTRAN modules contain an 
average of 35-45 statements. Assembly language modules 
have an average length of 50-60 statements. 

Each activity and module is uniquely identified by a six 
character code. Examples are shown in parentheses in Fig¬ 
ure 1. Thus module RVW128 would be module 128 in the 
View (VW) Process of the Routing (R) Subsystem. 

Structured design, a la Constantine, is applied either at 
the process or activity level, depending on the complexity 
of the process and constituent activities. Design documen¬ 
tation always includes a structure chart (hand drawn, as 
generated in the design process) and a completed module 
description form (MDF) for each and every module. A mod¬ 
ule description completely defines the function of a module 
so that it can be coded from this documentation alone, with 
perhaps reference to appropriate data structure documen¬ 
tation. 

For an easily grasped function, e.g., the binary search of 
a single precision array, a description of the calling sequence 
may be adequate documentation. However, it has been our 
experience that many modules require either flow charts or 
pseudocode for unambiguous documentation. This is true 
even though the emphasis is on the function not the detailed 
coding of the module. Typically, this situation obtains for 
the higher level modules in a structure chart. At that level 
it is frequently the case that the function of the module and 
the flow of control within the module are not really sepa¬ 
rable. 


RESULTS OF APPLYING STRUCTURED DESIGN 

Productivity figures 

The advanced planning activity implemented 336 modules 
of applications support software. These modules contained 
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Advance 

Group 

(336 Modules) 

Main Develop¬ 
ment Effort 
(2200 Modules) 

Man Hours 

Per Module 
Specification 

3.3 

5 

Structured 

3.0 

5.4 

Design 

Coding and 

8.8 

9.8 

Integration 

Total 

15 

20 


Figure 2—Manpower statistics 


11,104 lines of code, an average of 33 lines/module. There 
were 180 FORTRAN modules (average length 25 lines), and 
156 assembly language modules (average length 42 lines). 

The manpower figures were specification and analysis, 28 
MW (man weeks); structured/implementation design, 25 
MW; and code and integration, 74 MW. Figure 2 displays 
the corresponding figures per module. 

The largest activity in this collection of software contained 
117 modules (3551 lines), of which 105 (2936 lines) were 
FORTRAN. The distribution of calendar time for this pro¬ 
gram was: specification and analysis, 26 weeks; structured 
design, 6 weeks; and coding and integration, 20 weeks. 
Thus, although specification and analysis only accounted for 
22 percent of the manpower it consumed 50 percent of the 
calendar time. 

Column 3 of Figure 2 summarizes the figures for the bulk 
of the CAD software. This data represents approximately 18 
man years of effort including supervision time. Accounted 
for are 27 processes, 53 activities, and over 2200 modules. 

Problems encountered 

Although our experience with structured programming has 
been strongly positive, we did encounter some problems. 
Moreover, looking back, we see areas where improvement 
is needed. 

The biggest need for improvement has to be in the area 
of specification. Once a good specification has been gener¬ 
ated things become very manageable. However, we have 
found it extremely difficult to write a specification which on 
the one hand a user can read and understand, and which on 
the other hand defines things well enough to allow design to 
begin in earnest. 

A second problem area is related to one of the design 
goals. From the start it was impressed upon the program¬ 
mers that they were to put design before efficiency. As a 
result a couple of activities were implemented which were 
very much more expensive to execute than they should have 
been. These subsequently, had to be modified for efficiency. 

In most cases this efficiency problem might well have 
been avoided had we more strictly followed one of our own 
published procedures. We did not require each module to 
be reviewed by another programmer as we said we would. 
The excuse tends to be “we just don’t have the time”, and 
is used by programmers and supervision alike. In the face 


of tight schedules, this excuse is not easily dismissed. This 
problem will doubtless be struggled with for some time to 
come. 

A final problem is the difficulty in training enough pro¬ 
grammers to be good designers. The training problem is 
particularly troublesome because there is no way to give 
years of experience, and the attendant design maturity, to 
even a highly capable junior individual. This is important 
because the structured design of a large process requires the 
judgment to make numerous decisions, which involve trade¬ 
offs between strict adherence to structured design princi¬ 
ples, on the one hand, and effective use of human and 
machine resources on the other. The best solution is to 
employ the most qualified designers on critical designs, and 
to use less critical designs for training. But when schedule 
constraints force many important designs to overlap, less 
desirable compromises have to be made. 

Summary of benefits 

Although we did not have any solid prior productivity 
data, we definitely feel that structured design has improved 
productivity. This, however, was not a goal in our adopting 
structured design, but has been just a happy side-effect. 
What was anticipated were increased reliability, increased 
maintainability, and increased visibility. 

There is no doubt that we have achieved increased visi¬ 
bility. Moreover subsequent experience with modifying 
structured programs convinces us that increased reliability 
and maintainability have been achieved. 

It was found that structured design allowed us to make 
very good use of personnel. We have been able to really 
load up an activity in the module coding and checkout 
phases without introducing confusion. It has also been easy 
to quickly move a programmer from one activity to another, 
with little loss in effectivity. Moreover, we have gotten 
excellent productivity from beginning programmers. 

Only mild reluctance to adopt structured design tech¬ 
niques was manifested by the staff. Junior personnel adopted 
easily with no apparent “loss of individuality” response. 
There was some initial thrashing with senior personnel as 
we all strove to understand the implications and tradeoffs of 
modularity. For instance, not everyone accepted at first that 
structured design was indeed distinct from what they were 
already doing. 

A final word on interpreting our experiences with struc¬ 
tured design. We feel that our experience is unique to our 
application and our environment. A different application in 
a different environment might yield better or poorer results. 
Nonetheless, we are confident that, for most applications, 
structured design will yield more reliable and maintainable 
systems, while providing good visibility of the design and 
implementation processes. 
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EXPERIENCE WITH EXXON’S IMPLEMENTATION 
OF THE JACKSON PROGRAM DESIGN METHOD 
—C. M. Bernstein 

In 1973, Exxon’s Mathematics, Computers and Systems 
Department conducted an evaluation of the new technolo¬ 
gies of program development. The project was motivated by 
the increasing manpower cost for software development and 
maintenance and the increasing business vulnerability to 
software failure. We concluded that a program’s structure 
is the key to its effective development, enhancement, exe¬ 
cution and support. Without a program’s structure we could 
not reliably construct the program in parts, put the parts 
together such that their interactions would be predictable, 
and have the whole structure achieve its specified purpose. 

Michael Jackson, now of Michael Jackson Systems Ltd., 
was then an ACM lecturer. We found that he had a teachable 
method for the logical design of structured programs and a 
precise notation to express them. In addition, his method 
was effective at attacking practical programming problems 
where the designer is not free to define input and output 
formats and user interfaces. Michael taught three program 
design courses at our Florham Park, New Jersey offices in 
the winter of 1973. The courses were well received and, 
together with Michael’s consulting on specific applications, 
provided the knowledge we required. We adapted Jackson’s 
Methdology to our environment and renamed it Program 
Structure Technology, PST. 

It is important to remember that PST does not address the 
systems design process. PST is not concerned with defining 
the organization or contents of files, specification of input 
and output formats or transactions, designing data bases, or 
defining required processing. PST is concerned with the 
work of designing and implementing the program which 
meets those specifications. PST incorporates the technolo¬ 
gies of top down design, structured programming, top down 
development and test, and structured walk-thrus. 

We conducted our first in-house PST course in August of 
1974 and aggressively fostered the assimilation of PST over 
the following three years. Over 1,000 programmer/analysts 
have been taught PST. All Exxon regions can now provide 
their own training and support. This includes training for 
supervisors, consulting, and the support of standards. PST 
is being used in both large and small installations to develop 
batch commercial and interactive data base applications, as 
well as minicomputer, process control and program product 
software. 

The Jackson Program Design Method defines hierarchical 
program structures whose components are dissected. There 
are four basic component types: 

Elementary, which are not dissected 

Sequence, whose parts are executed once each, in order 


Selection, one of whose parts is executed, the choice part 
depending on a condition 

Iteration, which has only one part which is executed zero 
or more times 

The four component types have exact analogies in data 
usage structures for files, records and in internal data. The 
program’s structure is designed to correspond to the usage 
of the data to be processed. Alternative usages of data can 
be imposed on a given set of data and the alternative pro¬ 
gram structures evaluated. Each operation carried out by 
the program appears in an appropriate component. For ex¬ 
ample, “INITIALIZE CUSTOMER TOTAL’’ should ap¬ 
pear in a component which happens once per customer. If 
no appropriate component exists, the program structure is 
deficient. Where there is a conflict between data usages to 
be processed by one program, the program is designed as if 
it were two or more separate programs. The separate pro¬ 
grams are subsequently combined by the technique of pro¬ 
gram inversion. Where a data usage cannot be handled sim¬ 
ply, more elaborate forms of iteration and selection must be 
used. 

The major benefits of the Jackson Methodology are as 
follows: 

Reduced program complexity 

Eliminated logic errors at design instead of debugging in 
the testing stage or later 

Identified sensitive points in the problem specification 

Easily maintained programs 

Effective program documentation as a by-product of de¬ 
sign 

Step-by-step methodology has enabled effective use of 
software to support the process 

Four PST projects of varying size (.5, 1, 5.1, and 25 work 
years of effort) were evaluated and productivity was found 
to be above 7000 lines per work year in every case. This 
compares favorably with the New York Times Archives 
project’s 9000/WY and the “Industry Averages” of 2000- 
4000 lines/WY. In the case where we attempted to measure 
program quality, we found less than one error per program 
during the first six months of production. 

We have done little to the Jackson Methodology to adapt 
it to the Exxon environment other than minor changes in 
terminology and emphasis. We have complemented it with 
our own material on structured programming in PL/I, top 
down development and test, and structured walk-thrus. We 
have also employed different pedagogical techniques to en¬ 
able programmers who are not experienced instructors to 
teach the PST course. 

The Jackson Methodology has been extremely successful 
in Exxon and is now virtually a standard throughout the 
world. I recommend not underestimating the difficulty of 
teaching programmers “how to program properly” and de¬ 
velop an assimilation plan and a skilled staff to accomplish 
it. 
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INITIAL EXPERIENCE WITH A METHODOLOGY 

FOR CORRECT PROGRAM DESIGN—F. T. Baker 

In the forthcoming book, Structured Programming: The¬ 
ory and Practice, 1 the authors describe three techniques 
which have been incorporated into a Methodology for 
achieving correct designs. These are: 

1. A view of program correctness as a demonstration of 
a correspondence between the function of a program 
design (i.e., the set of ordered pairs corresponding to 
input states and output states) and the function re¬ 
quired by its specification. This approach, when used 
with stepwise refinement, permits selective and incre¬ 
mental correctness proofs to be carried out, since it 
incorporates a procedure for verifying the correctness 
of the expansion of a specification into any one of a 
basic set of control structures. (The expansion of a 
specification at any level into a program design can 
thus be verified, contingent on the correctness of 
lower-level specifications and their expansions.) Fur¬ 
thermore, proofs can be carried out with varying levels 
of rigor, ranging from a set of questions the designer 
may use to validate an expansion to a formal demon¬ 
stration recorded in a precise manner. 

2. A method for incorporating specifications into program 
designs to support correctness demonstrations when 
desirable. Each specification (either initial, or those 
generated in the expansion process) is retained as a 
comment (logical commentary) directly associated with 
the control structure which refines it. 

3. A design language (Process Design Language) to assist 
in the design process and to record the history of a 
design. PDL includes a standard “outer syntax” of 
essential control and data structures, and encourages 
development of an “inner syntax” appropriate to each 
design environment. 

Figure 1 is an example of a design for a program which is 
to save the maximum value occurring in an input sequence. 
In that design, each of the paired brackets encloses logical 
commentary which is a functional specification for the con¬ 
trol structure which follows it (sequence, ifthenelse or while 
do in this case). For each of the structures the appropriate 
proof procedure can be carried out to demonstrate the cor¬ 
respondence between its specification function and its pro¬ 
gram function. Furthermore, this can be done prior to the 
completion of the expansion process, or even on selected 
portions of the design. 

The methodology is primarily aimed at the detailed design 
of a program. It covers the period between the formulation 


[m:=maximum(inputseq)] 
do 

[(inputseq= empty— >m:= undefined | 

inputseq ^ empty ->m: = next( inputseq)] 
if 

inputseq=empty 

then 

m:= undefined 

else 

m: =next (inputseq) 

[m:=maximum(m,remainder of inputseq)] 
while 

inputseq # empty 

do [m:=maximum(m,ngxt(input)] 
temp:=next(input) 

[m:=maximum (m,temp)J 
if 

temp>m 

then 

m:=temp 

fi. 

od 

od 

Figure 1 


of an effective system design, and the translation from the 
design language into an implementation language. It was 
developed to introduce more precision into the design proc¬ 
ess and to encourage more consistent expression of designs. 
Whether or not formal correctness demonstrations are car¬ 
ried out, the stress on viewing programs as functions, de¬ 
veloped from specifications through a rigorous refinement 
process, should help achieve the goal. 

Experience to date suggests that the methodology is ca¬ 
pable of being practiced in the application development en¬ 
vironment. We believe that the control structures inherent 
in PDL are sufficient to support all levels of the design 
process, from system and module specification down to 
precise algorithms. The invention and use of logical com¬ 
mentary direct attention to specification and program func¬ 
tions, as they were intended to do. In particular, they en¬ 
courage the designer to specify and deal with boundary 
conditions and anomalies which frequently are poorly at¬ 
tended to and which sometimes lead to difficult-to-find er¬ 
rors. Finally, the view that each progrm should be designed 
as if it is to be proved correct, means that even if correctness 
demonstrations are not formally carried out, the program 
has a greater likelihood of properly embodying its specifi¬ 
cation. 

Experience to date also indicates several areas where 
more work is needed. The nature of the expansion process, 
and the desire to record the design history, mean that much 
copying is done as the design is developed. An interactive 
support tool appears useful to assist designers in this expan¬ 
sion and recording. The data structures in PDL (stacks, 
queues, sequences and sets) are useful ones, but better proof 
techniques to validate operations with them must be devel¬ 
oped. There is a notational problem inherent in specifying 
functions precisely, particularly in nonmathematical envi- 
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ronments, which must be solved through a combination of 
inventing better notations and abstracting operations. Fi¬ 
nally, the ability to demonstrate correctness does not mean 
that it is appropriate to do so in all cases; better guidelines 
for applying the varying degrees of rigor possible in the 
methodology must be developed. 
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EXPERIENCE WITH THE IPAD SOFTWARE 

DEVELOPMENT METHODOLOGY—Susan Voigt 

NASA is sponsoring the development of a computer-aided 
design system for use by the aerospace industry. The sys¬ 
tem, denoted IPAD, is being designed and implemented by 
Boeing Commercial Airplane Company and Boeing Com¬ 
puter Services. IPAD is a software system to enhance the 
computer complexes of aerospace companies to improve 
speed, efficiency and reliability of the design process for 
complex aerospace vehicles. The contract calls for appli¬ 
cation of an effective software engineering approach to min¬ 
imize programming and software design errors, as well as to 
produce highly portable software. 

THE TECHNIQUE 

NASA established general guidelines for phases of the 
development and release of software in stages, to allow early 
user testing and experience. The development phases are: 

1. Definition of the Problem (namely the Aerospace de¬ 
sign process) 

2. Requirements Definition (integrated information proc¬ 
essing and functional requirements) 

3. Development of IPAD System Specifications 

4. Preliminary Design 

5. Detailed Design, Code, and Test for each incremental 
release 

6. Acceptance Test and Demonstration with sample prob¬ 
lems for each release 

7. User Training and feedback 

8. Software maintenance during remainder of develop¬ 
ment contract 

Several plans and specific documents were called for in 
order to encourage a systematic approach and a well docu¬ 
mented product: 

1. Management and Technical Plans to describe general 
approach 

2. Configuration Control Plan to control and track all 
changes to requirements, design, code and documents 


3. User Involement Plan to insure the sytem developed 
is satisfactory to the users 

4. Test Plans to establish systematic procedures for de¬ 
velopment and acceptance testing 

5. Software Standards Handbook 

6. Requirements and Preliminary Design Documents 

7. Preliminary User’s Manual written during design so 
early user feedback can be obtained 

8. User and Demonstration Manuals 

9. Installation and Maintenance Manuals 

As the basis for the IPAD software engineering method¬ 
ology, the Boeing IPAD Development Team selected the 
Boeing Computer Services “Systematic Software Develop¬ 
ment and Maintenance (SSDM)” approach. SSDM is basi¬ 
cally a set of general guidelines for all phases of the software 
life cycle, and it corresponds well with the NASA require¬ 
ments. 

An Industry Technical Advisory Board (ITAB) was es¬ 
tablished at the start of the contract to closely involve the 
prospective user community. They have helped review and 
critique the requirements definition and software design 
phases. Subsequently, they will have the opportunity to 
install and test the software at their own computing facilities. 


EXPERIENCE AND EVALUATION 

At the time of this writing, the development is in phase 4, 
Preliminary Design. The techniques used to date and an 
assessment of their usefulness in the software development 
process is described below. 

In phase 1, a reference aircraft design process was doc¬ 
umented in flow diagrams indicating activities and decision 
points, with accompanying discussions. Also communica¬ 
tions between various disciplinary groups participating in an 
aircraft design project were diagrammed to illustrate the 
complex network of interfaces. Separate volumes were writ¬ 
ten to document the interactions between designers and the 
manufacturing organization and the activities in managing a 
product development. These three volumes written by en¬ 
gineers representing potential users defined the problem to 
be addressed with the IPAD system. 

The requirements were defined by the engineers in phase 
2. The BCS technique SAMM (Systematic Activity Model¬ 
ing Method) (Reference 1) was used to chart the inputs and 
outputs (description and quantity) for each activity in the 
flow-charts of phase 1. The user’s view of his requirements 
for computer-aided support in the aerospace design process 
also was documented. 

The results of phases 1 and 2 represent a very thorough 
definition of the aerospace design process and CAD users’ 
needs. These were well-received by the user community and 
have provided guidelines for their own analyses within their 
respective companies. The flow diagrams of the design proc¬ 
ess and the SAMM charts of the data flow are well correlated 
and provide a very systematic look at the problem. 
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Phase 3 was done by the software team, assisted by the 
engineers. They developed a concise set of IPAD require¬ 
ments based upon the engineering documents and NASA 
requirements for the IPAD system. A formal analysis 
checked that each requirement was complete, correct, un¬ 
ambiguous, precise, consistent, relevant, testable, traceable, 
free of unwarranted detail, and manageable. The engineering 
team developed criteria for acceptance testing for each re¬ 
quirement. Each test criteria was summarized in a paragraph 
which accompanies the requirement statement in the IPAD 
Requirements Document. The requirements were reviewed 
carefully by both NASA and ITAB, with considerable feed¬ 
back and revision resulting. A set of IPAD system specifi¬ 
cations were not produced, per se; the IPAD Requirements 
became the baseline for further development. 

The formal analysis of requirements was not successful. 
Ambiguities, inconsistencies, and redundancies were very 
difficult to eliminate, especially between similar require¬ 
ments. These arose from using secondary sources in devel¬ 
oping the IPAD requirements and giving them weight equal 
to the primary source, the engineering definition of the prob¬ 
lem. A satisfactory set of requirements was obtained through 
careful reorganization and joint review by software and en¬ 
gineering team members and NASA. The inclusion of the 
summary for acceptance test was very helpful in clarifying 
the intent of a requirement, as well as setting the stage for 
later testing. It also forced the engineering and software 


teams to collaborate, and is highly recommended for future 
projects. 

In Preliminary Design, phase 4, a user interface model 
was developed using state transition diagrams (Reference 
2). This included a set of user functions correlated to the 
IPAD requirements. These state diagrams have been used 
to walk through “user scenarios” to illustrate the function¬ 
ing of IPAD from a user point of view in performing a 
specified set of tasks. The other major components of IPAD 
are: the executive, the information processor, and the other 
systems interface and are currently undergoing design by 
separate subteams. Coordination among the various sub¬ 
teams in producing an integrated design has been difficult. 

While the approach used has been successful in achieving 
a good set of requirements, IPAD is not yet developed and 
the design methodology is unproven. The basic concepts 
appear to be an effective approach and further assessments 
can be made when the software is developed. 
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Formal methods in programming and microprogramm ing 


Two of the three sessions in this area are about microprogramming, and the 
third is about programming in general. The dominant theme in all the sessions is 
the current attempt by researchers and practitioners to make programming a 
formal process. Roughly speaking, this means the use of mathematical rigor in 
defining, specifying, translating, transforming, analyzing and verifying programs 
(including microprograms). 

The work is motivated not by a love of formal methods for their own sake, but 
by hard economic objectives. Users are now demanding increased confidence in 
the adequacy and reliability of delivered software, greatly reduced cost of pro¬ 
duction and maintenance, and greatly increased flexibility with respect to change, 
growth and transportability. These demands are not being met by the informal 
techniques that have accumulated over the years. The work on formal methods 
is intended to make major advances in the way software practice meets these 
demands. 

With few exceptions, the old techniques—and the tools that support them, 
have aimed to help the programmer to describe, encode, recode, package and 
probe his large and complex product. The task of fully understanding the product 
and of ensuring that it and all the steps used in its production are correct, has 
been left to the human judgement of producers and consumers. With some serious 
exceptions, present practice has been generally successful due to the dedication 
of computer professionals, the effectiveness of social processes in dealing with 
errors in output data, and, of course, the great economic benefit of using com¬ 
puters for solving problems. This success has come at high costs that are no 
longer acceptable. Furthermore, new applications and market conditions have 
arisen that bring very high penalties for error, inflexibility and inefficiency. 

The response of the computer science community has been to attempt to put 
mathematical rigor into the task of understanding what a program does (its 
“semantics”), or, in linguistic terms, what is the meaning of its text. 

Of course, understanding precisely what a program does implies knowing 
precisely also what the translators, optimizers and interpreters do that may 
operate on the program. If the description and analysis of programs and their 
processors could be put on a firm mathematical basis, and if there were effective 
procedures for carrying out formal analyses, the consequences would be very 
significant. For example, one could build powerful optimizers, and be confident 
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that they would not change the meaning of a program. Also, one could verify 
that a program implemented a requirement for all possible values of data in the 
intended input domain, without having to use test data samples (a most costly 
and unsatisfactory process if high confidence is demanded). 

As long as ten years ago, theoretical results showed that a formal basis for 
programming could be constructed. Since then much effort has gone into creating 
methods of formal specification and analysis, and tools to implement the methods, 
that are effective enough to be incorporated in software engineering practice. At 
the present time, several methodologies are reported to be in practical use in 
special application domains such as microprogramming, or on an experimental 
basis, for general programming. The methodologies require skilled practitioners. 
Because of this, one cannot say that a methodology is “ready” or “not ready” 
without saying who is going to use it. Nevertheless, it is claimed that there will 
be enough practitioners of the requisite skills to make a very significant economic 
impact on software production, especially as the methodologies mature. 

The papers and panel sessions of this area give the attendees an opportunity 
to judge how valid these claims are, and to assess the relevance of the ideas and 
tools of the new methodologies to their own practice. 




An approach to firmware engineering* 

by DAVID A. PATTERSON 

University of California 
Berkeley, California 


INTRODUCTION 

Although microprogramming has only been a research topic 
for the past 10 years, the concept is nearly as old as com¬ 
puters. 1 Wilkes originally defined microprogramming as a 
systematic and orderly approach to the design of the control 
section of a computer. Rather than using an ad hoc approach 
with counters and decoders to generate control signals, 
Wilkes proposed that the control be organized as a matrix. 
This memory like approach to control remained largely of 
academic interest until the late 1950s. The first large-scale 
microprogrammed computer was the IBM 7950 in 1961. 
However, it was the introduction of the IBM System/360 
line (with all but the largest system microprogrammed) that 
revived interest in microprogramming. The advance in speed 
and the reduction in cost of read-only memory (ROM) tech¬ 
nology transformed microprogramming from an academic 
concept of mild theoretical interest to a practical approach 
to the design of commercial computers. 2 IBM was able to 
offer several internally different models with the same in¬ 
struction set in order to provide a spectrum of cost and 
speed through the use of microprogramming. With the suc¬ 
cess of the 360, several microprogrammed computers were 
and are being built. In the late 60’s some manufacturers 
made microprogramming available to the users. Nearly all 
computer manufacturers have at least one computer imple¬ 
mented via microprogramming, and the expanding list of 
manufacturers that provide user microprogrammable com¬ 
puters include Burroughs (E1700), Control Data Corporaton 
(Cyber-17), Data General (Eclipse), Digital Equipment Cor¬ 
poration (PDP-11/60) and Hewlett Packard (HP-21 MX). This 
has expanded the potential domain of microprogrammers 
from computer architects to systems and applications pro¬ 
grammers. 

While microprogramming and “ordinary” programming 
have existed for almost the same amount of time, little of 
the considerable research in programming languages and 
methodologies has impacted the microprogramming field. 
Unfortunately, the following statement is a fairly accurate 
description of the state of the microprogramming: 3 

“At present, microprogramming is an elite activity, per- 


* This research was supported in part by the Department of Energy under 
Contract EY-76-S-03-0034, PA214. 


formed effectively only by a small number of expert prac¬ 
titioners. The work is detailed, precise, time consuming, 
and considerably more expensive than present-day soft¬ 
ware programming. But, computer manufacturers have 
found they can get dramatic improvements in system per¬ 
formance by converting software into microprogrammed 
form.” 

An approach to expanding the group of practitioners of 
microprogramming is to reduce the complexity and expense 
of firmware by providing tools similar to those that have 
reduced the cost of software. The Strum system was de¬ 
signed as a prototype to explore the benefits and detriments 
of such an approach. 

SOFTWARE ENGINEERING TECHNIQUES 

The most obvious technique to enhance the production of 
microprograms is to provide a high level microprogramming 
language (HLML). One reason HLMLs have not been used 
is the importance of efficient firmware. With restricted size 
and execution times being primary characteristics of micro¬ 
programming, some doubt whether a HLML could ever be 
useful. 15 With the newer computers control memory has ex¬ 
panded (e.g., 16K control memory of the HP-21mx) thereby 
relieving some of the concern about size. A more important 
aspect is the influx of applications microprogrammers who 
have a very different set of constraints from the micropro¬ 
grammer of the native instruction set. In the latter case one 
may be able to justify doubling the effort to reduce execution 
time by 10 percent. In the former case it is likely that ease 
of use, development cost and maintainability are much more 
important than small gains in execution speed. The necessity 
of a high level microprogramming language is further illus¬ 
trated by the Figure 1. It is widely accepted that the pro¬ 
grammer spends half his time testing the program. This 
figure indicates that a microprogrammer may spend half his 
time coding the microprogram. As it is widely accepted that 
a high level language will reduce coding time it makes emi¬ 
nent sense to provide a high level microprogramming lan¬ 
guage if it can produce reasonably efficient code. 

Another technique that has reduced the time to develop 
software is structured programming. This phrase is so widely 
used that the criteria that determines whether a program is 
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Figure 1—Percentage of time spent in specification, coding and test phases 
of microprogram and program development (from Reference 4). 


structured is vague. By structured programming we mean a 
disciplined approach to programming that starts with the 
highest level of abstraction and proceeds with a topdown 
hierarchical development of levels of modules that are suc¬ 
cessively refined and expanded until the bottom refinement 
is the program in some programming language. A key aspect 
of this approach is for every module at each level to be 
“intellectually manageable,” i.e., restrict the size and flow 
of control to create modules that are easy to read and to 
understand. Although assembly level programming does not 
preclude structured programming, it is easier to follow these 
concepts if the programming language supports this meth¬ 
odology. We believe that a cleverly designed microprogram 
language can hide the “unstructured” aspects of micropro¬ 
gramming plus provide tools conducive to structured micro¬ 
programming. 

A final aspect of software engineering is validation. Here 
an attempt is made to determine whether a program is “cor¬ 
rect.” Unfortunately testing does not guarantee that a pro¬ 
gram is correct; it only means that someone (probably the 
creator of the program) is satisfied that the program pre¬ 
forms properly for some subset of the total input domain. 
As Dijkstra states: 5 

“Program testing can be used to show the presence of 

bugs, but never their absence!” 

If we conclude that testing is inadequate, our only alterna¬ 
tive is formal program verification. This work began with 
the inductive assertion method of Floyd 6 that was elaborated 
by Hoare and Manna. This method first requires the addition 
of redundant text called assertions. Assertions are nonexe¬ 
cutable comments that describe the desired state of the 
variables of the program. These assertions can then be me¬ 
chanically combined with the program text to produce log¬ 
ical formulas. If proven true, these formulas show a kind of 
consistency, usually called partial correctness, between the 
text and the assertions. 

It is not clear that program verification is a practical tool 
for validating microprograms. In fact, there is a great deal 
of skepticism of the practicality of proofs of correctness in 
any environment. 7 The criticism is that only “toy” programs 
are proved, i.e., programs that use only integers or vectors 
of integers, deal with numeric algorithms (e.g., division), 
and are relatively short. Using these measures, one would 
have to conclude that microprograms are “toy” programs. 
Microprograms deal with integers (usually two’s comple¬ 
ment binary numbers) or integer vectors (memories), deal 
with numeric algorithms (instruction sets), and are relatively 


short (frequently less than 1024 words). Thus, the applica¬ 
tion of correctness proof techniques to microprograms may 
prove beneficial. 


STRUM SYSTEM 

The prototype system developed to test this approach to 
firmware engineering is called Strum (from STRUctured 
Microprogramming). This system includes a Pascal-like high 
level language, an optimizing compiler that produces micro¬ 
code for the Burrough’s D-machine, and a verification sys¬ 
tem. 4,8 The Burroughs D-machine is a general purpose mi- 
croprogrammable, multiprocessor computer. 9 The computer 
was designed by the Burroughs Advanced Development 
Organization in 1968-1969 to be used in future Burroughs 
systems. The D-machine has several innovations. It was 
designed to be used in multiprocessor configurations. The 
arithmetic section of the machine is expandable in 8-bit 
increments from 8 bits through 64 bits. It was the first 
machine to use two levels of control memory (micromemory 
and nanomemory) to try to obtain the advantages of hori¬ 
zontal and vertical microprogramming. And at the same time 
it was built, it was one of the few computers to offer a 
control memory that could be altered by the user during 
program execution. The D-machine allows up to 4096 56-bit 
microinstructions. 


THE EXPERIMENT 

As a meaningful test of the system we decided it was 
essential to emulate a real computer. If a paper computer 
were to be emulated one might wonder whether the design 
of the new instruction set had not been compromised in 
order to avoid difficulties with the high level language or to 
simplify the verification of the emulation. Although this test 
would be emulation of a real computer on a real micropro- 
grammable machine, some would wonder whether program 
verification, high level languages, and structured program¬ 
ming would not lead to correct yet inefficient code. As speed 
and size are very important in the microprogramming envi¬ 
ronment, it is possible that the overhead incurred using such 
techniques makes this approach unusable in the “real 
world.” 

To overcome these possible objections, an emulation of 
the same real computer was done twice. The first version 
was implemented via techniques traditionally used with mi¬ 
crocode on the D-machine, i.e., assembly level coding, de¬ 
bugging the microcode on the machine, and testing the mi¬ 
crocode using programs written for the machine being 
emulated. The other version was written using techniques 
proposed in this paper. Unfortunately it is difficult to create 
tests that are free from the appearance of bias. If one person 
did both emulations, one could correctly argue that the ex¬ 
perience gained on the first emulation project would improve 
the results of the second project. To avoid this criticism, a 
different person did each project. The argument, then, is 
that with two people the differences in education, experi- 
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ence, motivation, etc. affect the accuracy of the results and 
possibly affect the fairness of the comparison. One could 
also claim that the choice of a minicomputer for the exper¬ 
iment rather than a large-scale computer does not allow 
generalization of the results.* We can only respond that 
both microprogrammers are bright, capable people, that 
there are fewer and fewer distinctions between minicom¬ 
puters and large-scale computers, and that we can only hope 
that the reader will appreciate our attempt to make a fair 
experiment. The test case was the emulation of the HP-2115 
computer. 

The Hewlett-Packard HP-2115 computer 

The Hewlett-Packard HP-2115 is a predecessor of the HP- 
2100 and HP-21MX computers currently marketed by Hew¬ 
lett-Packard. The HP-2115 has 70 basic instructions that 
operate on two accumulators, two flags, and (in this emu¬ 
lation) 8192 words of 16-bit main memory. This instruction 
set includes several special input-output instructions for up 
to 64 devices and provides for multi-level indirect and paged 
addressing. The HP-2115 will handle up to 64 different in¬ 
terrupts. Although the Hewlett-Packard manual lists 70 basic 
instructions, most of the opcodes in the shift-rotate group 
and alter-skip group can be combined to form more than 100 
additional useful instructions. 

HP-2115 emulation using traditional techniques 

The emulation using traditional techniques was imple¬ 
mented by a graduate student in Spring 1975. This micro¬ 
program was written in TRANSLANG, the assembly lan¬ 
guage for the D-machine. He did not attempt to use the 
techniques of structured programming. He debugged the 
microprogram directly on the D-machine by running HP- 
2115 programs and examining the registers at each micro¬ 
step. As we indicated above, this primitive approach is un¬ 
fortunately typical of the way most microprograms are de¬ 
veloped. This student had ideal conditions for using this 
approach in creating the emulation: 

(1) He was thoroughly familiar with the HP-2115. He had 
spent two years as an M.I.T. undergraduate using this 
computer. 

(2) He was thoroughly familiar with the D-machine. This 
student was in charge of maintaining the D-machine 
and the implementing enhancements to it. 

(3) He was the sole user of the D-machine during that 
period. Usually a microprogrammer must compete 
with other users for access to the machine. 

(4) The hardware was fixed and operational. With com¬ 
puter manufacturers the computer is usually being 
built and debugged while the microprogram is being 


* The selection of which computer to emulate was made by the person who 
made the emulation using traditional techniques, not by the author. It should 
also be mentioned that the language was designed before the computer was 
selected. 


debugged.t It is often difficult to distinguish between 
an error in the microprogram and an error in the hard¬ 
ware. With formal microprogram verification, the 
hardware is not needed until after the microprogram 
is verified, i.e., much later in the development cycle. 

(5) There was software available to test his program. 
When a new instruction set is created, there is usually 
no program available to test the microprogram. 

After he was satisfied that his microprogram worked, we 
tried running diagnostic programs for the HP-2115 on the 
emulation. Diagnostic programs are used to test if the hard¬ 
ware is working properly. This is a widely used technique 
to test accuracy of emulation. The argument is that if the 
diagnostics believed that the HP-2115 “hardware” was 
working correctly, the emulation of the HP-2115 must be 
working correctly. Running the diagnostics uncovered two 
errors in this emulation. 

Subsequently we discovered that one of the unassigned 
opcodes performed a useful function. This opcode was ob¬ 
viously not tested by the diagnostics, but its use was appar¬ 
ently so widespread by users of the HP-2115 that they de¬ 
fined this opcode in a later programming manual. We were 
also surprised that the diagnostics did not test some appar¬ 
ently unusual combinations of options in the I/O instruc¬ 
tions. The student left this out of his emulation because he 
doubted that it would ever be used. (In March 1976 he 
completed the emulation of all unusual I/O instruction op¬ 
tions.) In writing the specification for the STRUM HP-2115 
emulation, another error was discovered. Memory reference 
instructions can use the current page address to supply a 
portion of the operand address. When an interrupt occurs 
the program address register is not modified but the com¬ 
puter selects and executes an instruction from the first 64 


TABLE I.—A Comparison of Size and Speed of Microassembly, 
Unoptimized Strum, and Optimized Strum Emulations of the HP-2115 
Computer 




Unoptimized 

Optimized 


Microassembly 

Strum 

Strum 

Number 




of Micro¬ 
instructions* 

946 

1124 

940 

Number 




of Nano¬ 
instructions* 

162 

123 

161 

Size in bits* 
Average number 

25504 

25856 

25344 

of cycles 
executed per 
instruction** 

72.9 

66.4 

58.6 


* The Burroughs D-machine has two levels of control memory so the size 
should be compared either in bits or number of microinstructions. 

** The Burrough D-machine comes in several microcycle times from 50 to 
500 nanoseconds. 


t A referee mentioned that usually a microprogram simulator is available 
while the hardware is under construction. Our experience is that the simulator 
and its corresponding hardware quickly diverge. While the simulator is useful 
this divergence combined with time constraints often force the microprogram¬ 
mers to have access to the computer before the hardware is debugged. 
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locations in memory. It was not clear from Hewlett-Packard 
documentation whether a memory reference instruction that 
requested address modification from the current page would 
get the information from the program address register or 
from the current page of the instruction (page 0). This em¬ 
ulation used the program address register but a test of an¬ 
other Hewlett-Packard computer revealed that the address 
bits always came from page 0. These three examples show 
the danger in relying on diagnostic programs as the only 
method of verifying an emulation. This emulation used 946 
microinstructions which required 162 unique nanoinstruc¬ 
tions. The execution speed is shown in Table I. The values 
in these tables are given in terms of D-machine microcycles. 
The D-machine at UCLA runs at 800 nanoseconds per mi¬ 
crocycle although Burroughs has developed versions that 
run as fast as 50 nanoseconds per microcycle. 

Strum HP-2115 emulation 

The second emulation of the HP-2115 using a high level 
language, structured programming and formal program ver¬ 
ification was implemented in Spring 1976 by the author. We 
had serious reservations about this experiment, mainly be¬ 
cause this microprogram would be the first microprogram 
that used the STRUM system. A simpler test case to catch 
any bugs in the STRUM compiler and program verification 
system was planned, but lack of time ruled this out. The 


result of the output from the optimized code is shown in 
Table I. 

Testing the HP-2115 emulation 

The testing of the STRUM HP-2115 emulation was per¬ 
formed on November 11, 1976 by the creator of the other 
emulation and the author. The test consisted of executing 
the HP-2115 diagnostic routines. These routines exhaus¬ 
tively test all opcodes and most register values for the mem¬ 
ory reference, alter skip, shift rotate and overflow instruc¬ 
tions. We did not know what to expect, because this was 
the first test of the STRUM compiler, and the first large- 
scale test of the STRUM verification system. The verifica¬ 
tion process uncovered 30 errors in the emulator divided 
equally between the executable microprogram, the specifi¬ 
cations, and the assertions. The program and verification 
formulae were corrected by hand. Although one large (about 
2000 lines of NUCLEUS code) program has been verified, 10 
this program was never executed. We were in the uncom¬ 
fortable position of using untested software tools to verify 
a program that would immediately be subjected to exhaus¬ 
tive testing. 

Surprisingly no errors in the STRUM emulation were un¬ 
covered by the HP-2115 diagnostic programs. Four errors 
were found in the code generated by the compiler, but had 
we run a simple compiler test program beforehand, all these 



Figure 2—STRUM HP-2115 microprogram hierarchy (1 means one page in 
listing; 2 means 2 pages in listing) 
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errors would have been detected. This was indeed a gratify¬ 
ing result. To the best of our knowledge the STRUM emu¬ 
lator is the largest formally verified and tested program. 

Summary and results 

Although all the results were gratifying, we did not expect 
the STRUM version of the emulator to be faster than the 
TRANSLANG version. Many believe that high level lan¬ 
guage microprogramming is too inefficient to be useful. 3 
Possibly an explanation for the unexpected result is the 
difference between “microprogramming-in-the-small” and 
“microprogramming-in-the-large.” For small examples (10 
to 20 microinstructions) it is feasible for a microprogrammer 
to spend hours optimizing microcode. However, with a 1000 
instruction microprogram, it is extremely difficult to consis¬ 
tently convolute one’s thinking to write efficient microcode. 
Also, high level languages and hierarchical structure make 
it easier to perform the global optimizations that may save 
more time or code than any instruction-by-instruction optimi¬ 
zation. Perhaps the most exciting result was the success of 
the verification of microprogram. Although working with 
only a prototype system, we were able to verify a complete 
emulation of a real computer on a real microprogrammable 
computer. Moreover, although the exhaustive testing per¬ 
formed by the diagnostic programs did find four bugs in 
compiler, it did not reveal any errors in the STRUM emu¬ 
lation. Not only did the verification uncover program errors, 
it uncovered errors in the specification and assertions. No 
matter how practical testing may seem, it will not uncover 
errors in documentation (specification) nor in the comments 
(assertions). The microprogramming language STRUM 
proved to be very easy to use. The use of macros and good 
control structures resulted in a well-structured micropro¬ 
gram. Figure 2 shows the structure of the HP-2115 emula¬ 
tion. Each of the 14 boxes corresponds to a functional sec¬ 
tion of the emulator. Only two sections occupied more than 
one page of the listing and 12 of the 14 boxes are macros 
rather than procedures. STRUM object code and the original 


HP-2115 EMULATIONS 

ORIGINAL 

STRUM OBJECT TRANSLANG 



STRUM SOURCE 

CODE 

EMULATION 

LOOP’S 

2 

42 

11 

GO TO’S 

0 

144 

144 

LABELS 

0 

166 

123 

CALLS 

16 

37 

104 


Figure 3—Frequency of use of statements which make programs more 
complex 


microassembly emulation contained many more jumps, la¬ 
bels and called statements than the STRUM source program 
(see Figure 3). The STRUM source program also contained 
fewer loops because the “handshaking” firmware used to 
access memory includes loops that are generated by the 
STRUM compiler. 

CONCLUSION 

With respect to the microprogramming field we believe this 
experiment has demonstrated the validity of the use of each 
facet of the STRUM approach: use of a high-level micro¬ 
programming language, structured microprogramming, and 
formal microprogram verification. While we believe it is 
most effective to use all facets of the STRUM approach, the 
reader should be forewarned that the development of such 
system is a tremendous undertaking. The development of 
production versions of an efficient optimizing compiler and 
verification system is probably a five to ten man year effort. 
Hopefully tools will be created which will simplify this task. 

Readers interested in a more detailed analysis should see 
References 4, 8, 11 and 12. 
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Code optimization techniques 
for micro-code compilers 

by C. J. TAN 

IBM T. J. Watson Research Center 
Computer Sciences Department 
Yorktown Heights, N.Y. 


INTRODUCTION 

With the ever expanding volume of system functions directly 
implemented in microcode and the growth of microprocessor 
applications, it has become necessary to design high level 
language compilers for these machines to achieve high pro¬ 
gramming productivity. However, because of the very low 
level architecture of many of these machines, compilers that 
generate efficient code for these machines have not been 
produced. 

An optimizing compiler, called PL/MP, has been designed 
which is capable of supporting a variety of microprocessors 
such as the Motorola M6800 as well as machines like the 
IBM 370 model 145, and several microprocessors designed 
and used internally at IBM. 1 PL/MP uses a set of novel, 
machine dependent, and the more conventional high level, 
machine independent optimization techniques to achieve 
high object code quality. Experimental results have indi¬ 
cated that the machine dependent optimizations play a very 
significant role in enhancing the quality of the object code 
produced by the compiler. The most effective of these op¬ 
timizations are 

1. Register allocation: optimized binding of compiler 
source and temporary variables to fast machine regis¬ 
ters. 

2. Low level addressing code optimization: code motion 
and common expression elimination for addressing 
code. 

3. Code consolidation: combining a set of scattered sim¬ 
ple instructions into a single more complex instruction 
supported by the object machine. 

Basic to all of these optimization techniques are the al¬ 
gorithms for obtaining information concerning the use of 
variables throughout the program. These data flow analysis 
algorithms, 2,4,5 are used throughout the compiler for both 
machine independent and machine dependent optimizations. 
In this paper we will describe the code consolidation proc¬ 
ess, the environment in which it is carried out, and its 
dependence upon global data flow analysis. Algorithms for 


register allocation, code motion and hoisting have been pub¬ 
lished elsewhere, 2 ’ 3 ’ 6 and will not be discussed in this paper. 


THE PL/MP COMPILER 

The PL/MP compiler accepts a high level language such 
as PL/I and generates efficient object code for a micro¬ 
machine. The compiling process takes the source code 
through several levels of internal text as shown in Figure l. 1 
Note that the front end of the compiler is concerned with 
the machine independent transformations and optimizations, 
while the back end is concerned with the machine dependent 
transformations. The interface between them is the A-text. 
Operands in A-text are generic, symbolic registers of arbi¬ 
trary length. 

Frequently, micro-machines have a large number of fast 
registers. Even though these registers often have inhomo¬ 
geneous usage constraints, it is important to optimize the 
use of these resources. Hence an inherent design philosophy 
in PL/MP is to assume, early in the front end, a machine 
with an indefinite set of registers. Source level variables are 
loaded into the register space and kept there. Their values 
as well as those for the compiler generated temporaries are 
stored into memory only if absolutely necessary. The traffic 
between memory and register space is thus kept to a mini¬ 
mum. 

The internal A-text 7 is basically a register transfer lan¬ 
guage. However, it is general enough to support a large class 
of high level languages and low level micro-machines. It 
consists of primitive and complex operations, which may or 
may not be all supported by a given object machine. 

Operands in A-text are constants, labels or generic reg¬ 
isters. All computational operations are register-to-register. 
The only operations addressing memory space are loads and 
stores. Thus all addressing code has been exposed at this 
level. A sample list of primitive and complex A-text units is 
shown below. 

Note that the complex A-text operations, such as add 
indirect, load indirect, etc., can all be replaced by a se¬ 
quence of equivalent primitive operations. For most ma- 
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Figure 1—PL/MP structure 


chines an add-indirect-with-offset, for instance, will be more 
cost effective than the equivalent set of primitive instruc¬ 
tions. However, if the variables are used at several places 
in the program then under certain conditions many primitive 
addressing instructions can be optimized out by code motion 
or hoisting . 2 


THE ENVIRONMENT FOR CODE CONSOLIDATION 

Assume that we have a machine with a set of general 
registers Rj through R 16 such that only Ri through R 4 can be 
used as memory address registers. We further assume that 
the machine supports indirect addressing with and without 
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TABLE I.—Primitive and Complex A-text Operations 


Operator 

Operand 

Comments 

Primitive Operations 

ADD Ri ,R -2 ,R{i 

add registers 2 and 3 and store result in 1 

ADDC 

R 1 .R 2 .R 3 

add with carry 

BLE 

L,Rj ,R-2 

branch to L if R, less than or equal to R 2 

LOAD 

Ri.p 

load into R, from location p 

NOT 

R 1 .R 2 

store into Ri one’s complement of R 2 

CALL 

L 

call subroutine L 


Complex Operations 


LOADO 

Ri.P.n 

load with offset n[Rj<-MEM(p+n)] 

LOADIO 

Ri.R2.n 

load with offset indirectlR,<-MEM((R 2 )+n)] 

LOADI 

R 1 .R 2 

load indirect 

ADDI 

Rj.R 2 .R 3 

add indirect 

ADDIM 

Ri.p.k 

add immediate[Rj<-p+constant k] 

ADDIO 

R 1 .R 2 .R 3 jn 

add indirect with offset 


offsets. Thus, in this respect the target machine is at a higher 
semantic level than that of the primitive A-text. 

Figure 2 shows a segment of an example in which we 
want to add a variable v to an array A of length 100 bytes. 
Both A and v are stored in a data area whose origin is at 
address p. The array’s first element coincides with the area’s 
origin and variable v is stored at an address 100 bytes beyond 
this origin. Thus, as shown in the figure, we first load into 
a register @a the address of array A. At program point p4 
the array element is then loaded into register x through the 
load indirect instruction. The variable v is similarly loaded 


into register y, and it is added to x and stored in register z. 
Finally, the contents of z is stored back into the array, and 
the array element address @a is incremented by one to get 
to the next element. After iterating through the loop 100 
times the program branches to point p7 where we proceed 
to other parts of the overall program. Figure 3 shows, in 
primitive A-text, the program after addressing code optimi¬ 
zation by code commoning and moving the addressing code 
outside of the loop. 

Notice that the sequence of code (p2,p3) which loads the 
variable v into the symbolic register y can be combined into 
the single instruction ‘load indirect with offset’, i.e., 

LOADIO y,@b,100 [R y <-MEM((@b)+100)] 

Once the above consolidation is performed the code se¬ 
quence (p2,p3,p5) can be combined, so resulting, at p5, in a 
single add-indirect-with-offset instruction 

ADDIO z,x,@b,100 

Figure 4 shows the sample program after code consolida¬ 
tions at p2, p3, p5, and p7. 

Notice that in the above consolidation process we have 
eliminated the register y containing the value of the variable 
v. Hence these consolidations are effective only because the 
symbolic register y, itself, is not referenced elsewhere in the 
program. Consider a slightly different situation, shown in 
Figure 5, where the register y is used at point p8. The 
subtract immediate instruction involves the register y and 
cannot be consolidated with the addressing code for y. This 
implies that we must keep the value of v in this registery. 
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pQ: 1 LOAD o»a,p | 
pi: | LOAD Ob, p I 
p2: \ ADDIft b,Ob,100| 
p3: | LOAD1 y,b I 




Figure 3—Sample program in A-text 
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-Sample program in A-text 




Therefore, the instruction first consolidated from the se¬ 
quence at (p2,p3), i.e., 

LOADIO y,b,100 

cannot be deleted in the next consolidation iteration. Hence, 
for this case, replacing the code at p5 and p7 with the more 
complex instructions would actually increase the object code 
size. 

It is clear from the above example that the decision to 
perform code consolidation can be made intelligently only 
after examining the global usages of the variables in ques¬ 
tion. It is not surprising, therefore, that the compiler strategy 
is to first generate A-text with only primitive instructions. 
The primitive A-text, with addressing code fully exposed, 
is then subjected to global code optimizations such as com- 
moning and code motion. Finally, the possibility of code 
consolidation is explored to generate an equivalent A-text 
using complex instructions where advantageous. 

CODE CONSOLIDATION ALGORITHM 

We have seen, in the previous section, instances where 
scattered code sequences can be consolidated. We will now 
describe the procedure for performing code consolidations. 
The procedure involves: (1) recognizing a possible consoli¬ 
dation candidate, and (2) performing the actual consolidation 
if it is profitable to do so. 


Code pattern classification 

For many representative machines, the primitive A-text 
code patterns that correspond to complex instructions con¬ 
sist of an addressing code sequence followed by a compu¬ 
tational operator, which we will call the terminal operator. 
Furthermore, it is often possible to find a set of terminal 
operators that share an identical addressing code sequence. 
Sequences having this commonality are grouped together to 
form a pattern class. For instance, the code sequence p2, 
p3 in Figure 3 is a member of the class that contains indirect- 
with-offset operations such as LOADIO, STOREIO, and so 
forth. Similarly, the sequence at p3, p5 is a member of the 
indirect operations class. 

For many machines there exists a partial ordering among 
the code pattern classes. For example the code sequence 
(p2,p3,p5) corresponds to the complex operation ADDIO. 
Notice that the sequence (p2,p3) forms the complex opera¬ 
tion LOADIO, and similarly (p3,p5) forms the operation 
ADDI. Hence the arith-indirect-with-offset(AIO) class is 
greater than the class containing LOADIO as well as the 
class containing ADDI. 

The patterns for our fictitious machine may be classified 
as below, where the first instruction in the sequence is called 
the ‘token’, and the primitive instructions are divided into 
arithmetic, memory-register classes, and so forth. 

From this classification we can derive the substitution 
rules for these patterns as shown in TABLE III. 
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TABLE II.—Pattern Classes 


Class 

Instructions 

Token 

Terminal code 

Cl 

(mem-reg-ind-off)= 

ADDIM 

(mem-reg-ind}= 


{LOADIO,..} 


{LOADI,STOREI,..} 

C2 

(arith-ind)= 

LOADI 

(arithmetic )= 


{ADDI.SUBI,..} 


{ADD,SUB,..} 

C3 

(arith-ind-off)= 

LOADIO 

(arithmetic }= 


{ADDIO,...} 

ADDIM 

(Cl) 


That is, the instruction in C3 (e.g., ADDIO) replaces 
either the sequence consisting of LOADIO followed by the 
corresponding arithmetic instruction (e.g., ADD) or the se¬ 
quence of ADDIM and a sequence from C2 (e.g., ADDI). 
Notice that all tokens are specific instructions either primi¬ 
tive or complex. Furthermore, a code sequence in a sec¬ 
ondary pattern class, such as C3, can always be formed 
from sequences of a smaller class. It is obvious that if no 
code pattern in Cl or C2 can be profitably consolidated, for 
instance, then no pattern in C3 can be profitably consoli¬ 
dated either. Hence it suffices to perform consolidation 
starting from the smallest element in the partial ordering 
defined by the pattern classes. 

Code sequence recognition 

Before we can make a judicious consolidation decision 
based on relative costs, for a given A-text and a specific 
pattern class C, we must first find a valid code sequence in 
that class. This involves (1) finding a sequence of code 
satisfying the substitution rules, and (2) checking that the 
operands in this sequence satisfy certain data flow con¬ 
straints. 

Notice that, starting from the token of a code sequence, 
each operation defines a new symbolic register. This defined 
value, unaltered, must be used in the immediate successor 
operation in the sequence. Furthermore, the target variable 
at the token, called the substitution variable, must not be 
altered along the path containing the code sequence so that 
it can be substituted at the terminal operation. 

Since the elements of a sequence may be scattered 
throughout the program, it is a difficult task to check for 
these conditions. Fortunately, very efficient global data flow 
techniques have been developed 5 that enable us to generate 
certain global relations among program variables. We will 
now introduce some of these data flow concepts, and then 
show how they can be used to find a valid code sequence 
for a given class. 


TABLE III—Substitution Rules for Pattern Classes 
<C3>:: =LOADIO<Arith)| ADDIM<C2) 

(C2) : =LOADI {Arith) 

<C1> := =ADDIM(Mem-reg-ind> 


Let (v,p) denote a variable v defined at program point p. 
We say (v,p) reaches point q if there is a path from p to q 
such that v is not redefined along that path. If q is a program 
point with two or more predecessor statements then we say 
(v,p) is available at q if (v,p) reaches q through every one 
of its predecessor statements. Availability implies that if v 
is also defined at another point, say r, then (v,r) cannot also 
reach q. Since the reach and availability information can be 
determined efficiently for any program, we can state the 
following 

Variable Flow Condition: if p* and p 3 are two consecutive 
code points in a code sequence then the variable defined 
at pi, as well as the substitution variable for the sequence, 
must be available at pj. 

We have already mentioned that every substitution se¬ 
quence starts with a known token. Hence having found a 
token for a given class we must start scanning the text to 
find the next code point in the sequence that satisfies the 
variable flow condition. The following constraints allow us 
to search through the program rather efficiently. First we 
will introduce two additional data flow concepts. 

The first concept we need is the so called variable live¬ 
ness. If a variable v defined at p reaches a point q and is 
such that there is a subsequent use of (v,p) in a path starting 
at q then we say (v,p) is live at q, otherwise we say (v,p) is 
dead at q. Thus we need not search along a path where the 
variable being searched is either dead or not available. 

The second concept we need is that of node dominance. 
(For the following discussions we use the term node to 
denote a block of straight line code in a program. Hence 
p0,pl,p2,p3 in Figure 5 constitute a node. Similarly, p5,p6 
is a node.) We say a node d in a program flow graph dom¬ 
inates another node n if every path from the program entry 
node to node n goes through node d. For a given program 
a dominator tree can be obtained such that there is an edge 
connecting nodes d and n if and only if d dominates n. For 
the program in Figure 3 the dominator tree is simply the 
graph minus the loop edge from p6 to p4. 

Notice that if (pi,p 2 ,P 3 , . . . ,pi) is a substitution sequence 
then in order to replace this sequence at point pi by a more 
complex instruction, the node containing p x must dominate 
that containing p 2 , and so forth. Therefore, starting from the 
token at Pi we need only follow the path defined by the 
dominator tree. Fortunately, for any flow graph a dominator 
tree can be obtained very efficiently as a part of the general 
data flow analysis. 2,5 
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The overall algorithm 

Once we have found a valid substitution sequence the 
next question is whether or not it is advantageous to actually 
consolidate the code. Notice that if (pi,p 2 , . . . ,p0 is a sub¬ 
stitution sequence, then we can always replace the operation 
at Pi by the complex operation that is logically equivalent to 
the function of the sequence. However, since the replace¬ 
ment operation is usually more complex than the original 
operation at p l5 we presume it is advantageous to consolidate 
only if all the code at the other points in the sequence can 
be deleted. This is possible only if the intermediate variables 
are all used by terminal operations of the same class and 
found to be valid for consolidation. Otherwise, we cannot 
delete them, and hence there may be no advantage in re¬ 
placing the original operation at p ( by a more complex one. 

Starting with the token we search for all possible substi¬ 
tution sequences of the given class. If a code sequence is 
found to violate either the variable flow condition, or one of 
the intermediate variables is used in a statement not allowed 
by the sequence substitution rules, then the intermediate 
addressing code cannot be deleted. Hence no code consol¬ 
idation will take place and the process terminates. 

Note that if a code pattern is greater than another pattern 
in the pattern class ordering, as is the case with C3>C2 and 
C3>C1 in our previous example, then in searching for a 
sequence for C3 we must encounter a consolidated text unit 
in either class Cl or C2. Otherwise, since we always start 
the consolidation process for the smaller class first, this 
implies that no code can be advantageously consolidated for 
the patterns in Cl and C2. Hence no code pattern in class 
C3 can be consolidated either. In fact, the substitution rules 
for C3, as shown in Table III, clearly indicate that a valid 
sequence in that class must contain either a Cl or a C2 
operation. 

The above discussion can be summarized by the follow¬ 
ing: 

Algorithm C 

Input: Program expressed in A-text, program flow graph, 
variable liveness, availability, and dominator tree, 
pattern classes and substitution roles. 

Output: Consolidated A-text. 

Method: 

1 . Starting from the smallest pattern class C that has not 
been previously processed, perform the following 
steps. If no unprocessed class is left then terminate. 

2. If consolidation failed for all lesser classes of C then 
go to step 1, otherwise proceed to step 3. 

3. Starting from the root of the dominator tree, look for 
a previously unprocessed token of the given class. If 
a token is found then proceed to step 4, otherwise go 
to step 1. 

4. Search for substitution sequences in class C along the 
paths defined by the dominator tree. 


5. If no substitution sequence is found or if the variable 
flow condition is violated then return to step 1. 

6 . If a use of an intermediate variable is made by an 
operation not part of a code sequence then go to step 
1 . 

7. Replace terminal operations found by appropriate op¬ 
erations in class C, and delete the intermediate ad¬ 
dressing code. Return to step 3. 

CONCLUSIONS 

We have discussed some of the techniques used for machine 
dependent code optimizations for a micro-program compiler. 
One of these techniques, namely code consolidation, has 
been discussed in detail. It should be obvious that the com¬ 
pile-time efficiency of these optimizations is heavily de¬ 
pendent upon the efficiency of the global data flow analysis 
package used by the compiler. 

The code consolidation process, as described, may seem 
to be rather complex. However, for typical real machines, 
the number of pattern classes turns out to be rather small, 
and the substitution sequence length is also usually short. 
Hence with the use of a high-powered data flow package, 
the algorithm indeed can produce very efficient code with 
reasonable computation. 

Experiments conducted for one of the commercially avail¬ 
able micro-processors, namely the M6800, have shown that 
code consolidation together with addressing code optimi¬ 
zation have contributed a 30-40 percent improvement of the 
object code produced by the PL/MP compiler. 
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INTRODUCTION 

Microprograms have been around for some time now, and 
so have errors in them. Therefore techniques have been 
developed for finding the errors. We review these existing 
techniques and explain why we are working on a new one. 
The main advantage of our method is its ability not only to 
detect errors, but (if there are no errors) to give a formal 
proof that the microprogram does perform according to 
specifications. Using our approach we are working on actual 
machines and actual microcode. We discuss our experience 
with such proofs of correctness and what kinds of errors we 
are finding. The reader will also find a comparison of prob¬ 
lems in microprogramming with problems of general soft¬ 
ware reliability. 

WHY IS MICROPROGRAM VERIFICATION 

CONSIDERED NECESSARY? 

“Microprogramming Trend Considered Dangerous” is the 
title of a letter in the ACM Forum. 1 The letter continues, 
“Thus the development and maintenance process for micro¬ 
code will increasingly resemble that of systems and appli¬ 
cations programs. And therein lies mortal danger.” This 
paper will briefly trace the history of microprogramming and 
microprogramming support systems. As microprogramming 
has developed, better support systems have evolved so that 
microprogramming has been a useful tool. Now, with the 
widespread development of microprocessors on a chip, and 
use of read/write control stores, new ideas for support sys¬ 
tems must be developed. This paper then describes one such 
idea, symbolic simulation for microprogram verification, and 
indicates its necessity. The goal of this symbolic simulation 
of microprograms is to find all errors in the microcode (and 
correct them) while proving that the final code is correct. 
To do this, the Language for Symbolic Simulation (LSS) is 
used to prepare an APL-like executable description of the 
architecture (or program) specifications. The same language 
is used to prepare a description of the microprocessor. Then 
it is proved mathematically that the microprocessor when 
controlled by the actual microcode simulates the specifica¬ 
tions, using symbols for data instead of bits. This language 
and procedures are described, together with an automated 
set of program aids, the Microprogram Certification System 


(MCS). Some results are given of applying these techniques 
and MCS to real programs and computers. These results 
indicate that microprogram verification by symbolic simu¬ 
lation is feasible, and micro code maintenance problems 
indicate that verification is necessary. 

WHAT ARE PREVIOUS TECHNIQUES? 

When Wilkes first proposed microprogramming in 1953 
and built the EDS AC II, 2 computers were built entirely of 
discrete components. The store holding the microprogram 
was also built of discrete components and the microprogram 
was debugged like the rest of the computer—by graduate 
students. When hardware components were available so that 
microprograms became commercially feasible (in the late 
1950’s), read-only storage was preferred because of its speed 
and low cost. 3,4 Fixing errors in the microprogram then 
became slow and awkward, so special simulators were writ¬ 
ten to help debugging. Unfortunately every data flow or 
control flow change necessitated changes in both the simu¬ 
lation program and the microcode. A race developed— 
would the simulator or the computer be running first? 

However, computers controlled by microprograms have 
many convenient features. An architecture for a family of 
computers is desirable, but needs a complex instruction set 
for the top of the line and for ease of programming. 5 A rich 
instruction set costs very little more than a spartan set, if 
the computer is microprogrammed. 6 Rewriting application 
programs for new machines is expensive. But emulation of 
the previous computer instruction set by a new computer 
using the facilities of microprogramming allows the old pro¬ 
gram to run. 7 Maintenance of many distinct computers in a 
family is difficult, since the hardware for each computer is 
different. Again, using a microprogram allowed special 
maintenance techniques—microdiagnostics, scan-in, scan- 
out, log-out, instruction retry, etc. to be designed. 8 

This microprogram complexity doomed the individually 
written simulator, and more complex design automation sys¬ 
tems were written. One, the Controls Automation System, 9 
was written for the S/360 design effort. The computer hard¬ 
ware at the register transfer level was described by a For¬ 
tran-like programming language, including bit string han¬ 
dling. Design changes meant only a recompilation to produce 
an input for the simulator. The microprogram was written 
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at an assembly level, but with mnemonics for the microin¬ 
structions chosen by the design engineers. The assembled 
input to the simulator was kept in a program file, in which 
any single subroutine could be changed. The event driven 
simulator could be instructed to set up desired starting and 
ending for the simulation, to select the desired microcode 
for simulation, to produce snapshots of the registers at var¬ 
iable breakpoints, to dump in emergencies, to trace with or 
without timing checking, etc. This and similar systems 10 
were a definite advance, but problems remained. 


WHAT WAS WRONG WITH THEM? 

Another litany on the difficulties of program testing in the 
middle 1960’s is hardly needed. Anyone who has written a 
program to be used extensively by other people knows the 
problems. “Testing detects the presence of bugs, but never 
their absence” 11 is the well-known refrain. “Enough” test 
cases can never be generated in our limited time on earth. 
How is a good selection to be made? The coverage depends 
upon the skill and dedication of the individual who wrote 
the microprogram. But even if all paths are traversed by test 
cases, faults which are data dependent can still remain. 
Moreover, simulation in any guise is slow and expensive. 

Thus the systems did reach the field with faults in their 
control. Diagnosis of the computers themselves was diffi¬ 
cult. Most faults occurred many levels deep—a particular 
combination of instructions and data turned them into ob¬ 
servable errors. The best routines for exposing these faults 
were the higher level language compilers. (The diagnostics 
run, but the compiler doesn’t.) Therefore capturing the en¬ 
vironment so the fault could be discovered and fixed was 
difficult. A knowledge of many different design levels be¬ 
came necessary—from circuits (were race conditions trig¬ 
gered?) to logic design (did that decoder work correctly?) to 
microcode (was that transfer necessary?) to machine code 
(what series of instructions was traversed?) to the compiler 
(what statement is being compiled?). The well-known tech¬ 
nique of trying a fix to see how it worked could not be 
applied to microcode in read only storage. 


WHAT HAPPENED? 

Human ingenuity and perseverance triumphed again. Mi¬ 
croprogramming was—and is—a great success. The com¬ 
puters worked well, and the new facilities made available 
more answers per dollar spent. 

Hardware test support equipment 12 came into use—ex¬ 
pensive but necessary. Special computers, equipped with 
read/write control stores were built. Facilities for rapidly 
changing the control store contents were added, and fixes 
were performed in minutes. Changes could now be tried 
rapidly, and if they seemed to work all system and test 
programs could be run against the microprogram change 
before final commitment. Also, automatic data test genera¬ 
tors were built, and millions of pseudo-random data patterns 
could be processed. 


Now the bandwagon effect began and is still continuing. 
SIGMICRO (ACM) and the Technical Committee on Micro¬ 
programming (IEEE CS) were formed and began to hold 
their yearly workshops, presenting new ideas. If micropro¬ 
gramming is good for central processors it must be good for 
I/O equipment. 13 Look at all the new functions which can 
be performed! Need reliability/availability? Add special pro¬ 
gressive scan test routines. 14 Need help with communica¬ 
tions? Provide independent computing power. 15 Does the 
operating system take too much time and space? Put part of 
it in microcode. 16,17 Would you like a slightly different com¬ 
puter, or direct execution of higher level languages? Just 
provide the correct microcode. 18 Does your terminal take 
too much of your CPU time? Provide an intelligent terminal. 
All of these advances were fostered by improving hardware 
technology which provided more and more logic facilities in 
less and less space. Now (in the 1970’s) we have various 
types of microprocessors on a chip—CPU’s of various 
types, 19 I/O processors, 20 standard busses, and everyone 
(almost) can build his own distributed system. 

Help to do this is provided in abundance. Do you think 
that the use of higher level languages (Pascal, Algol, Fortran, 
etc.) will help your programming problems? Just provide the 
correct environment and the standard language can be used. 
Do you think that structured programming will help? 
Courses are readily available. Are you using your system 
for control? You can buy hardware test equipment which 
will simulate the environment in which your micro will run. 12 
Many advances in program testing are now available. Do 
you want a system which will analyze your program, and 
produce test inputs so that all paths will be traced at least 
once, and all instructions executed (if possible)? Such sys¬ 
tems are available. 21 Would you like to learn more about 
selecting test cases, devising adequate test procedures, ap¬ 
plying support systems to your own problems? Help is avail¬ 
able. 

So what’s the problem? Once again in our enthusiasm we 
are building faster and faster—and reliability is falling be¬ 
hind. Letters to the editor such as Lehman’s “Micropro¬ 
gramming Trend Considered Harmful” 1 point out the dan¬ 
gers in this. All of the techniques listed so far test specific 
cases—and the untested cases can still be erroneous. Mi¬ 
croprograms are hidden from all but the designer. Should 
we explain to the checker at the local supermarket that the 
buttons on the cash register send data to a computer which 
then uses this information to get a “correct” answer? Most 
of the time everything works correctly—but when errors 
occur rectifying them is expensive—and fixing the system 
is even more expensive. (Have you bought a very cheap 
pocket calculator recently?) 

WHAT ELSE IS AVAILABLE? 

The problems that microcode poses are also posed by 
ordinary programming systems when such systems are writ¬ 
ten to be used extensively by people distinct from the orig¬ 
inators. Ten years ago a formal, mathematically oriented 
approach, called program verification, was formulated. 22,23 
In this approach a series of assertions, written in first order 
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predicate calculus, are provided for the program input, out¬ 
put, and various intermediate points in the program. In par¬ 
ticular every loop must satisfy an assertion called a “loop 
invariant” specified by the user. Devising these invariant 
assertions is not easy, even though many papers have been 
written, each proposing a procedure which is effective in 
various special cases. 24 Various simple program modules 
(written in high level languages) have been proved correct, 
and more informal proofs have been given for larger exam¬ 
ples. 25 However, as with all proofs, the proofs are only as 
good as the hypotheses and the number of people who have 
actively checked them. 26 A difficulty with the pure form of 
this standard approach is that by starting from an externally 
given program text it necessarily loses sight of the program’s 
genetic origins. This deficiency has been recognized; 27,28 and 
proposals made to cure it. In general the idea has been to 
recreate, or use, a formal expression of the generating steps 
by which programmers construct programs in the first place. 
These proposals are still in the research stage. 

Another approach has been studied by King 29 and Clarke 30 
and experimental program aids have been constructed. They 
use as a major part of program validation the execution of 
the parts of the program between assertions using symbols, 
not the actual data values. The difficulty here (which is one 
of the difficulties in automatic program verification also) is 
defining and implementing the execution semantics for the 
symbols and the operators used in the programming lan¬ 
guage. 

Symbolic execution can be seen to have several deficien¬ 
cies. Symbol manipulation is costly and slow. Again the 
question of coverage arises. The data paths are represented 
symbolically, but which paths can be covered? Mostly main 
lines? What is the total cost? Will all errors be found? In 
Howden’s experiments 31 neither ordinary good testing tech¬ 
niques nor symbolic execution found all known errors in a 
Fortran program. In addition there are the real world prob¬ 
lems of the actual computer representation of data, with 
overflow, interrupts, etc. Another difficulty is that symbolic 
execution also ignores the knowledge that the programmer 
has about the specifications, and it does not check the spec¬ 
ifications. Recent studies have shown that wrong or inade¬ 
quate specifications account for 50 percent of program er¬ 
rors, and these errors are the most expensive to find. 32 

WHY DO THESE TECHNIQUES HAVE PROMISE 

FOR VERIFYING MICROCODE? 

The techniques of program verification have been applied 
for the most part to higher level language programs and have 
met with limited success. Fully automatic methods are al¬ 
most nonexistent—often a fairly detailed human analysis of 
the data structures and control structures used is necessary 
before automated aids can be applied. Also, such proof of 
necessity deals with some abstraction of what actually hap¬ 
pens in a computer when the compiled code produced from 
these programs is executed. Errors can therefore result be¬ 
cause the actual running program is not the program which 
was verified. Why should program verification techniques 
have any more promise when applied to microprograms, 


especially to actual microcoded implementations, than they 
have in other applications? 

As is described in the next section, our approach requires 
a complete formal description of the computer on which the 
microcode is to be run, and the actual microcode—the array 
of bit patterns—is used with a symbolic interpreter. Though 
this makes necessary a more complex description, because 
overflow, invalid array indexing, etc. must all be explicitly 
described, errors which would go undetected if these factors 
were ignored can be detected. 

The use of a single data structure—a two-dimensional 
array of bits—simplifies both the symbolic execution and 
the specification of the Floyd-type assertions or their coun¬ 
terparts. The registers, memory, I/O channels, switches, 
etc. of the computer are all represented as arrays. Also, the 
operations on these arrays are limited to storing and retriev¬ 
ing strings of bits addressed by indices, concatenation, re¬ 
shaping to other dimensions, and certain arithmetic and log¬ 
ical operations. Problems with program verification often 
arise when more complicated data structures are involved; 
if there are such structures in a microprogram, they are 
explicitly implemented in terms of the more primitive no¬ 
tions, and verification takes place over that more restricted 
domain. 

The fact that microprogrammers are not constrained to 
write in a “structured” way (and often do not in order to 
increase time or space efficiency) probably increases the 
difficulty of formulating and placing correctness assertions 
in microcode. However, once that is done, the control struc¬ 
ture of a machine description which interprets microcode is 
simple, and symbolic execution can proceed in a straight¬ 
forward manner. 

The way in which we use symbolic execution to prove 
microcode correct is another factor which improves our 
chances for success. As described in the next section, we 
do not place assertions at various points in the microcode 
which relate values of machine components to previous val¬ 
ues, or describe a state (e.g., sorted, permuted, etc.) which 
the computer must satisfy. Rather, we equate the machine 
components with components of a higher level description, 
a description of the architectural specifications which the 
microcode is supposed to implement. In this way we can 
partially avoid the necessity for quantifiers, single-program 
inductive invariants, and references to initial or previous 
values of machine components. 33 

The most compelling argument for the feasibility of mi¬ 
croprogram certification is our experience. We have de¬ 
signed and implemented an automated system to aid in ver¬ 
ifying microprograms, and have used it to detect and correct 
errors in “real” code. 

WHAT ARE THE DIFFERENCES BETWEEN 

VERIFYING MICROCODE AND VERIFYING 

HIGHER LEVEL PROGRAMS? 

A microprogram is just a special kind of program. The 
main difference between firmware and software is in the 
semantics of the programming language and the emphasis 
on the speed of execution for firmware. Microcode is inter- 
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preted by hardware, whereas higher level programs can eas¬ 
ily have their semantics defined in terms of another pro¬ 
gram—an interpreter. This makes a difference because a 
verifier must be given the semantics of the programming 
language. Thus for verifying microcode it is necessary to 
describe a piece of hardware as an interpreter. An inter¬ 
preter for a higher level programming language presents an 
idealized view of the environment (this is the whole idea of 
higher level languages). The lower we go towards the phys¬ 
ical realities the more we have to face to them; here are 
some of the realities we are faced with. 

Hardware has the inherent capability of operating in par¬ 
allel. Parallelism in software is certainly very common, but 
software can use the cushions provided by the lower layers 
of software. For example, memory interlocks prevent two 
processes from accessing a shared variable at the same time. 
Nobody will provide such interlocks for hardware, where 
shared variables are physical wires; therefore we have to 
watch for the possibility of races. Cooperating processes in 
software can take advantage of synchronizing primitives, 
which shield the software designer from the idiosyncrasies 
of hardware. No such shield is available for a microprogram, 
which is the shield for others. The microcode itself must 
handle such things as I/O, interrupts because of external 
causes, program-caused overflows, and other exceptions, 
which are “hidden” from the high level language program¬ 
mer and which most program verification techniques ignore. 

At the specification level, there are parts of the description 
which are left intentionally ambiguous, so that different 
models of a simple architecture (such as the 360 series) may 
implement them in different ways. In verifying a particular 
implementation, we must handle this ambiguity in such a 
way that it does not “lock in” the specification to a partic¬ 
ular implementation technique. 

On the other hand there are aspects of microprogramming, 
which make verification easier. Microprograms operate on 
one fixed data structure—bit vectors. Further, these oper¬ 
ations are not very sophisticated, in the sense that the bit 
vectors are not supposed to represent notions very different 
from themselves (at least as far as the architecture is con¬ 
cerned)—a bit vector typically represents a number. Thus 
the statement of correctness is usually close to what the 
microcode does, but the results are expressed in terms of 
numbers rather than bits. 

All these characteristics of microprograms make them 
good subjects of mechanical verification. They are so con¬ 
cerned with small details that it is difficult for people to 
examine them carefully. At the same time they are not 
sufficiently intellectually challenging for a person. But these 
are exactly the characteristics which machines find easy to 
handle, and therefore computers have a better chance in 
competition with people when verifying microcode than 
when verifying higher level programs. 

HOW SHOULD MICROCODE BE VERIFIED? 

Testing techniques are based on the ability to recognize 
an erroneous procedure. Verification proves that a program 
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written at a level above machine code is correct. What is 
needed is a system which will find all errors in a given 
program as it runs on a computer in a reasonable length of 
time. To do this three capabilities are necessary: formal 
specifications which are natural and easy to produce; cov¬ 
erage determination, i.e., the ability to recognize a correct 
procedure; and the ability to execute the code as existing in 
the microprocessor so that analysis of the results will deter¬ 
mine the fault. 

To formalize specifications we model all facets of system 
operation using the APL-like Language for Symbolic Sim¬ 
ulation (LSS). 34 This description is an abstract machine 
whose state consists of a facility vector describing the ma¬ 
chine (or storage) components and a control structure which 
permits parallel asynchronous operation. The facility vector 
components, two-dimensional arrays of bits, are described 
by dimension statements, as in Figure 1. These facilities are 
an 8 bit wide Main Storage with 2 24 locations, sixteen 32 bit 
General Purpose Registers and a 33 bit Accumulator. Op¬ 
erations upon the facility vector are specified by a library of 
macro routines written using APL-like operators, as in Fig¬ 
ure 2. 

This macro models a S/370 add register instruction. The 
2 ’s complement addition is modeled by decoding (-*-) the 
contents of the registers from bit strings to integers, adding 
the integers, then encoding (t) the result in 33 bits ((33p2)). 
This value is assigned (: =) to a. SETCOND is called, which 
sets the S/370 condition code. The low order 32 bits of a 
(bits 1 to 32, 0 origin) are put in GPR[R1;]. 

The macro SETCOND is shown in Figure 3. SETCOND 
shows the conditional structure of the LSS macros. OV- 
ERFLO is a predicate from the predicate library. If it is 
true, then bits 34 and 35 of the Program Status Word are 
both set to 1. Otherwise, if the result is zero, the condition 
code is (0 0), or positive (1 0), or finally negative (0 1). 

The last example is OVERFLO, which shows some of the 
logic operations available. Here, z is the 32-bit result of 
adding the 32-bit quantities X and Y (Figure 4). 

The LSS language is carefully defined in Reference 34. 
Only two kinds of objects are necessary for LSS, integers 
and two dimensional matrices of bits (truth values are one 
bit matrices). 

Modeling computing systems by describing their state 
vectors and the state transformations is intuitively plausible. 
LSS specifications are executable, i.e. given a state vector 


ADDREG (Rl, R2) = 

a: =(33,p2)- r (2-i-GPR[Rl;])+2-i-GPR[R2;] 
SETCOND (a,GPR[Rl;],GPR[R2;]> 
GPR[Rl;]:=a[l + i32] 

Figure 2 
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SETCOND(X,Y,Z)= 

OVERFLO(X,Y,Z)-»PSW[34+i2]: =(1 1) 

ELSE -*X[1]=0~* 

(X=32pO)->PSW[34+i2]: = <0 0) 

ELSE—*PSW[34+i2]: = (1 0) 

—>X[1] = 1-»PS W[34+12]: = (0 1> 

Figure 3 

value as input, a symbolic output state vector value may be 
obtained by symbolic execution. A symbolic value is an 
expression built from the operators of the LSS language and 
symbols representing the initial values of the variables, after 
appropriate simplification. For example, if A has the value 
(11110000) and B has a symbolic value $B, then the assign¬ 
ment statement A: = A&B becomes A: = (11110000)&$B; A 
is given the new value $B[ 4],(0000) (the here is caten¬ 
ation). The adequacy of the system model may be tested by 
executing cases of special interest. The model acts as a 
special purpose machine, i.e., it carries out one set of spec¬ 
ifications. 

LSS is also used to describe the hardware attributes of 
the computer on which the specified architecture or program 
is to be implemented by code. This implementation descrip¬ 
tion acts like a general purpose computer. When the assem¬ 
bled code to be verified (a matrix of bits) is inserted in 
memory, and a set of symbolic inputs is given, symbolic 
execution and simplification will determine the outputs. 

To determine if the desired action is identical to the action 
of the code, we use symbolic simulation. Heuristically 
speaking, the execution of the machine description and ac¬ 
tual code simulate the execution of the specifications if 
everything the specifications can do the machine and code 
can do, though possibly in a different way. Because the two 
facility vectors (specification and implementation) may have 
different components, to carry out the simulation it is first 
necessary to specify a set of relations between them. These 
simulation relation components specify, for selected pairs of 
specification and implementation control points, relations 
between the state vector quantities which must hold at these 
points. See Reference 35 for a complete and formal descrip¬ 
tion. 

To carry out the verification a symbolic execution tree, 
or proof tree, must be constructed. 36 An automated, inter¬ 
active system, MCS (Microprogram Certification System), 
has been designed and implemented to aid in this verification 
(see Figure 5). A node, or goal, in the tree represents a class 
of states of the system at one point in time. In general, a 
node will consist of 

(a) two state vectors—an assignment of symbolic values 
to facility variables; 

(b) two control points—positions on the control trees; 

(c) a predicate list—a list of conditions that the initial 
values must satisfy in order that the states represented 
by this node can be reached. 

OVERFLO(X,Y,Z)=(X[0],#X[1])&(Y[0]=Z[0]) 

Figure 4 
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Figure 5—Parts of the microprogram certification system 


The root of the tree (which represents the entire problem 
of proving the simulation) has a son, called a simulation 
relation node, for each of the simulation relation compo¬ 
nents. The state vector, control, and predicate lists of these 
nodes correspond to points at which the specified relations 
must hold. The values of the elements of the facility vectors 
in these simulation relation nodes satisfy the relationships 
necessary so that the implemented machine, under the con¬ 
trol of the program, will simulate the specification machine. 
A branch in the proof tree represents a computation path. 
The leaves of the proof tree represent all possible final 
states. 

The following process is used to build the proof tree. 
First, the simulation relation nodes are generated. In these 
nodes the most general symbolic values such that the con¬ 
ditions of the corresponding simulation relation component 
hold are given to the variables of the state vector. Any 
conditions which cannot be asserted in this way are placed 
on the predicate list. The next step is to run each abstract 
machine, performing symbolic computation until another 
corresponding pair of control points in the simulation rela¬ 
tion is reached. 

One machine is arbitrarily chosen to be run first. At this 
initial point variables have as values either symbolic con¬ 
stants (symbols representing unknown but fixed values) or 
values determined from the simulation relation. Control pro¬ 
ceeds as in normal execution. When symbolic constants are 
encountered in expressions being computed in assignment 
statements, the value assigned is a simplified combination 
involving operators and symbolic constants. 

When symbolic constants occur in predicates evaluated 
to determine possible branches, a single flow of control may 
not be able to be determined. The predicate may evaluate 
to a Boolean expression involving symbolic constants, such 
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as “$X=0,” which cannot be evaluated to “true” or 
“false.” In such a case all possible logically independent 
results of evaluating the predicate must be considered; in 
the previous case $X=0 and $X^0. The program doing the 
symbolic evaluation will generate a node (subgoal) for each 
independent result, add a predicate expressing the truth of 
the result to the predicate list, and simplify the result. In the 
case being considered, two subgoals will be generated. For 
$X=0, 0 will be substituted everywhere for $X; for 
$X=£0, this predicate will be put on the predicate list. The 
user then chooses the path he wishes to follow (the program 
is interactive) and begins again. The remaining paths will be 
traversed later. If the predicate involving symbolic constants 
evaluates to true or false, no path branching occurs. 

After generating a series of nodes and branches, the con¬ 
trol corresponding to a simulation relation component is 
reached. Then the other abstract machine is run until a 
simulation control point for that level is reached. 

Now the system verifies that the pair of control points 
reached defines the control component of a simulation re¬ 
lation node. The values of the two facility vectors are sub¬ 
stituted into the simulation conditions of this component, 
and a goal is generated for each simulation condition. Using 
the predicate list, a proof is attempted for each simulation 
condition. The process is repeated for each node of the tree 
which is not a final node. In the process a complete goal 
tree is formed. 

Newly created expressions are simplified at once to pre¬ 
vent the propagation of unsimplified forms. At present MCS 
has over 400 rules for the simplification of APL and logical 
expressions. If a theorem simplifies to “true,” then the goal 
is achieved (and theorems are the only goals generated in 
proving simulation that can be achieved directly, without 
generation of subgoals). If the theorem cannot be proved by 
the theorem prover, its simplified form is the theorem of the 
pattern in a single generated subgoal so that analysis by the 
user is facilitated. It may occur that a theorem cannot be 
proved, or that a pair of stopping points not corresponding 
to a simulation relation component is reached, or that a 
stopping point is not reached. Then an error must be sought 
in the code or in one of the descriptions. In addition the 
occurrence of an unexplained branch will signal the presence 
of an error. 37 

In MCS, theorem proving, which is simplification of APL- 
like expressions, occurs at a number of places and forms an 
important part of the system. 

This general process becomes more complicated when the 
asynchronous parallel operation of the system must be con¬ 
sidered. Our method of modelling parallelism involves con¬ 
sidering the processes as composed of Ks2 separate sub¬ 
processes running sequentially, but interleaving arbitrarily. 
No notion of time is built in. How the subprocesses inter¬ 
leave is immaterial as long as no shared variables are used, 
since these subprocesses are and remain independent. When 
shared variables are used, we have to consider which sub¬ 
processes can be executed between two references made by 
a single subprocess to a shared variable. We also must 
consider how the shared variables and the subprocesses 
interrelate in the process execution. To model this we pro¬ 


vide LSS operators to express when interleaving may occur; 
when interleaving is not specified a subprocess runs without 
interruption. 

A solution has been proposed by Brand and Joyner to 
consider the K separate subprocesses as each running with 
its own control tree and to add two statements to LSS, 
DELAY and WAIT, so that the shared variables can be 
correctly handled during the arbitrary interleaving of the 
subprocesses. 38 

EXPERIENCE USING SYMBOLIC SIMULATION 

The MCS is now being applied to a version of a real 
computer, the NASA Standard Spacebom Computer-2 
(NSSC-2). This version is called the Hybrid Technology 
Computer (HTC). This version was produced at the IBM 
Federal Systems Division at Huntsville, Alabama. 39 Its ar¬ 
chitectural specifications require it to support the 86 System/ 
360 Standard Instructions (no decimal or floating point op¬ 
erations). It has the usual sixteen 32 bit general purpose 
registers and 16 to 64K bytes of storage (with an extended 
protection mechanism). A single I/O channel supports three 
types of I/O: device initiated I/O, CPU initiated I/O and 
Direct Memory Access I/O. Device initiated I/O has two 
classes. Buffered I/O allows a device to transfer data to/ 
from a table in main memory without knowing the location 
or size of the table. External interrupt permits a device to 
interrupt the normal program sequence. CPU initiated I/O 
also provides two types of services: command and 16 bit 
data transfers to or from I/O devices or CPU to I/O 16 bit 
control operations. Direct Memory Access is transparent to 
the NSSC-2 microprogram controlled hardware unless a 
hardware error occurs, when the microprogram is called. 
Each use of the I/O interface, whether CPU or device ini¬ 
tiated, follows a certain protocol, depending on the I/O com¬ 
mand. 

Describing the NSSC-2 CPU was straightforward. The 
state vector facilities are essentially those listed in the S/360 
programming manual. The basic CPU macros are all of the 
type (and approximate length) of those described earlier in 
this paper. This control, however, involves recursive calls, 
so that instructions will be fetched and executed indefinitely, 
or until some interrupt or other external signal intervenes. 
The basic idea is shown in Figure 6. The macro which 
handles the interaction between the CPU and the I/O is 
much more complicated because of the asynchronous par¬ 
allel action of the CPU and I/O devices. This is carefully 
described in Reference 40. The description takes 25 double 
spaced pages of computer printout. 

This architecture is implemented by a data flow that has 
a 16 bit wide data path, three working 16 bit registers, a 16 

cpu = 

(SOFT-STOP= 0)—>DO-IN STRUCTION 

SERVICE-INTERRUPT 

CPU 

(SOFT-STOP=l) -> (1 
Figure 6 
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bit wide store whose storage data register does double duty, 
a 64x16 bit scratchpad memory, and a 16 bit I/O register. 39 
The data flow is controlled by a IK read only memory of 64 
bit wide horizontal micro-instructions, another 64x16 read 
only memory for computer instruction decoding, a 32 bit 
instruction register, and various sized registers and signals 
for speed of operation. The facility vector consists of the 
storage data flow registers and control signals. The macros 
describing the action of the 21 types of CPU oriented mi¬ 
crocommands are again straightforward. The I/O protocols 
depend basically upon 2 other microcommands. These pro¬ 
tocols require that certain acknowledgments be sent by 
the CPU, that the CPU wait until certain signals are received 
from devices, that interrupts not be ignored indefinitely, etc. 
Verification of these microprogram controlled protocols is 
an important part of microprogram certification (see Refer¬ 
ence 40 for a complete description). This description takes 
20 pages of computer printout. 

One part of the HTC microcode, where the new treatment 
of I/O was heavily used, is system reset. This is code exe¬ 
cuted upon start of the HTC, and its job is mainly to initialize 
various data structures. It also initializes the I/O channel 
status; here it is important to ensure that the protocols are 
strictly observed. During system reset the microcode uses 
a certain piece of code that is used also for other purposes. 
For these other purposes the request line (which can be 
raised by the I/O interface) is sensed. The operation of 
system reset, however, depends on this line being down. To 
ensure this the computer must prevent any device from 
being serviced by the I/O interface. Therefore before enter¬ 
ing the shared piece of code the microcode resets the inter¬ 



face. (That is, service of any device will be terminated). 
After resetting the interface the proper operation of reset is 
checked using this shared code. To prevent a device from 
getting service during this checking, the command line 
(under CPU control) is enabled. If these precautions were 
not taken, system reset would not be performed correctly 
when a device happens to request service at exactly the 
wrong moment. Needless to say, we could hardly expect 
this kind of error to be detected by testing. 

In the course of trying to prove correctness of system 
reset we found that, after checking that the reset was valid, 
the microcode jumped to a routine which implemented part 
of the protocol for sending the storage data register infor¬ 
mation to an I/O device. This is not only a violation of 
specifications, but the information transmitted is random— 
whatever happens to be in the storage data register when 
the power is turned on. Moveover, in this routine the mi¬ 
crocode sent an acknowledgment to the I/O interface with¬ 
out checking that the correct I/O routine had begun. If an 
unknown device were requesting service, it would receive 
the service acknowledgment even though no device input 
has been read. The device—and its input—would be actually 
unknown. After consulting with the author of the microcode 
it was determined that he used this feature for debugging 
purposes (indicating to Test Support Equipment that reset 
was finished) and had left it in the final product by error. 

After eliminating this jump to the routine which violated 
the architectural specifications, we were able to verify sys¬ 
tem reset, following 5 paths and proving 28 theorems. 

The next part of the microcode to be tested concerned the 
CPU IFETCH, 15 microinstructions, 3 paths and 45 theo¬ 
rems. Figure 7 gives an outline of the proof tree developed 
for this HTC microcode. In this part of the code we found 
that if a program overflow had been signaled, but upon return 
to the beginning of IFETCH an I/O interrupt was waiting, 
then the overflow would be ignored. We also found that no 
half-word instruction may reside in the last two bytes of 
memory. Since the memory contents are not available im¬ 
mediately, the first thing the IFETCH code does is fetch a 
word (16 bits) from memory, then immediately read the next 
word so 32 bits are read to fill the instruction register. If the 
instruction is 16 bits there is no slowdown, while if the 
instruction is 32 bits there is an increase in instruction speed. 
Unfortunately if the first 16 bits read are a 16 bit instruction 
in the last two bytes of memory, IFETCH tries to read the 
next two bytes, a memory overcapacity interrupt is given 
and IFETCH is never completed. This is an example of a 
bending of specifications, not a microprogram fault. A note 
in the programming manual would be better than slowing 
the HTC down. 

Another 97 microinstructions were executed during sym¬ 
bolic execution of 10 S/360 RR instructions. Thirty-two 
paths were followed, and 480 theorems proved. In the 
Branch and Link instruction (BALR) the link address was 
stored before the branch address was accessed. Thus BALR 
Ri Ri became a NO-OP instead of a branch and link. This 
was not detected by being unable to prove a theorem; in¬ 
stead, it was detected by generation of an unexpected branch 
of the goal tree. (The error had been found by further testing 
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before we found it.) One other specification exception was 
found: bit 31 in the key word in the Set Storage Keys (SSK) 
instruction must be zero. This is counter to the instruction 
description, which says that bit 31 is ignored. 

CONCLUSIONS 

Some architectures, data flows, programs and micropro¬ 
grams have been described using LSS. Microcode has been 
symbolically simulated and errors found. Modelling systems 
and programs using LSS has been relatively straightforward, 
except when trying to describe very efficiently asynchron¬ 
ous parallel actions. MCS has run with reasonable speed 
during these verification runs, and it appears that MCS can 
be used to verify microprograms of moderate size. Thus, 
the approach of symbolic simulation does offer some hope 
for correct microcode before customer shipment, and au¬ 
tomated methods have been successful in detecting errors 
in “real” microprograms, which would be very difficult or 
impossible to detect with program testing. Certification of 
microcode and machine level code may thus be the place 
where automated program verification techniques will reveal 
their applicability to actual software and firmware. 
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PANEL OVERVIEW—Karl N. Levitt 

Formal methods, i.e., use of mathematical rigor, have 
been employed by research computer scientists in their at¬ 
tempt to develop general results for many aspects of com¬ 
puter science, e.g., computational complexity, undecidabil¬ 
ity, numerical analysis, programming language semantics. 
Much of this work has had little impact on those charged 
with producing working software systems. However, in re¬ 
cent years numerous researchers have suggested that by 
applying formal methods to the realization of systems, the 
quality of such systems could be significantly improved. 
Such formal methods could be applied in the structuring, 
specification, verification, and analysis of performance for 
systems. The position statements below explore the use of 
these techniques in the production of systems. The general 
opinion is that formal methods will ultimately assume a vital 
role, but for the present their use will be restricted to par¬ 
ticular systems produced by skilled individuals. The use of 
the formal methods will gradually increase as the techniques 
are refined and applied to a larger variety of systems, as 
tools are developed to support their use, and as the general 
community becomes better educated in formal methods. 


FORMALISM CAN HELP YOU—Lawrence Robinson 

There is much resistance on the part of the programming 
community to the use of formalism in programming meth¬ 
ods. However, there is also a problem in achieving reliable 
and maintainable software that meets its requirements. De¬ 
spite the resistance, the application of formalism to pro¬ 
gramming techniques shows much promise in attaining a 
software product of high quality. At present, some advanced 
techniques are ready for limited use. 

First it is useful to state exactly what formalism is. For¬ 


malism is a means for making logical and consistent argu¬ 
ments about a particular area. To formalize an area requires 
a set of definitions and axioms to characterize the area, and 
a set of rules of inference to validate or disprove logical 
arguments. Formalism can be used to understand, analyze, 
and reformulate both a problem and its proposed solution, 
and to verify that the proposed solution actually solves the 
problem at hand. 

Other engineering disciplines make use of formalism to 
some extent (e.g., Laplace transforms in electrical engi¬ 
neering and partial differential equations in structural engi¬ 
neering). In fact, some aspects of software already make 
use of formal techniques, such as the exact specification of 
the syntax of a programming language. However, software 
engineering, because it is still a very young field, lags behind 
other engineering endeavors in the use of formal techniques. 

In addition, the special problems inherent in software 
engineering require an even greater use of formalism than 
do the problems of other engineering endeavors. Two factors 
are responsible for software engineering’s greater need for 
formalism: 

• The complexity of large software systems exceeds that 
of other engineering systems by several orders of mag¬ 
nitude. Any engineering practices that are developed 
for software must place considerable emphasis on the 
management of this complexity. Current software en¬ 
gineering methods have not adquately done so. Formal 
mathematics has been the classical tool for managing 
complexity, by combining many cases into canonical 
groups, and by structuring complex concepts by the use 
of abstraction and formal definition. 

• Software has a requirement for total exactness that is 
unique among engineering disciplines. Whereas the 
products in other fields are required to fall within con¬ 
tinuous tolerances, the smallest error in a piece of soft¬ 
ware (a single bit) will sometimes result in the cata¬ 
strophic failure of a system, rather than in degraded 
performance. Software engineering techniques must be 
able to guarantee the exact fulfillment of requirements. 
Formal techniques provide the only possibility of ex¬ 
actly stating the requirements of a piece of software, 
and for mathematically proving that the software meets 
its requirements. 
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Because software is so difficult an area, its development 
requires a great deal of attention to the analysis and planning 
that precede the coding. Current techniques underemphasize 
these early aspects of the software development process. 
Some new techniques require formal analyses of require¬ 
ments and formal design specifications before coding can 
proceed. These techniques are ready for use by advanced 
systems programmers. A methodology developed at SRI 
International (HDM—the SRI Hierarchical Development 
Methodology) has been used for formally specifying the 
designs of several operating systems and many intermediate¬ 
sized systems; some of the systems have been implemented. 

However, these techniques are not currently ready for 
everyone, and considerable training is required in order to 
use them. The major obstacle, however, is a resistance to 
change. But the education process (at universities, for ex¬ 
ample) is making these techniques available to people en¬ 
tering the software field. Gradually, programming will be¬ 
come closer to mathematics—more of a science than a black 
art. The changing process will be painful and fraught with 
resistance, but it hopefully will result in better and better 
software products. 


SOFTWARE DESIGN BY ALGEBRAIC 

SPECIFICATION—Ellis Horowtiz 

I’d like to adopt for the moment an extreme point of view 
so I can describe the range of application of algebraic spec¬ 
ification to software design. A recent advance in software 
construction has been the recognition that one should begin 
by supplying a formal specification of the intended task. 
This notion applies both to programming systems as well as 
to individual programs. The debate now centers on the way 
such a specification will be phrased, and as with program¬ 
ming languages practitioners are developing strong biases 
for and against certain approaches. My own bias centers on 
algebraic specification. 

Algebraic specification is not appropriate for specifying 
all possible tasks. For example, the technique does not seem 
natural for describing concurrency. What it does seem ideal¬ 
ly suited for is the description of data types (data struc¬ 
tures). Many computer applications can be conceived of as 
the transformation of data from one form into another. Then, 
the programming task becomes one in which the logical data 
structures must be precisely defined and then represented 
via other data structures which are available on a computer. 
The essential ingredient which makes the algebraic approach 
so successful for data type specification is the fact that a 
data type is composed of a set of related operations. These 
operations create, build-up or make smaller instances of the 
data type, they may decompose it into its constituent parts 
or answer questions about its form and content. All of these 
operations are best viewed together and algebraic specifi¬ 
cation naturally exposes these relationships. 

The relationships between operations of a data type are 
given via a set of equations, usually called algebraic axioms. 
These equations show the effect that the operations have on 


instances of the data type. One important virtue of these 
equations is that they imply no representation. Thus the 
meaning of the operations of a data type is defined abstractly 
(i.e., without regard to the eventual representation), and the 
design process is broken into two logical phases: data type 
definition and implementation. My own experience leads me 
to conclude that an algebraic axiomatization is about as easy 
to compose for a data type as it is to describe the type in 
English. 

The ease of specification might alone be sufficient to rec¬ 
ommend algebraic axioms. But it turns out that there are 
other payoffs when one adheres to this software design 
scheme. In particular the testing and verification of the re¬ 
sulting software are all facilitated when one continues to 
design and implement all data types via the algebraic disci¬ 
pline. A software “environment” can be created which aids 
the programmer as he builds his system. At the USC/Infor- 
mation Sciences Institute several of us (D. Musser, J. Gut- 
tag) have created the Data Type Verification System (DTVS) 
which supports program creation by algebraic specification. 

We begin by algebraically axiomatizing a single data type. 
Is our axiomatization correct? Before attempting a proof 
some testing would be desirable. DTVS will automatically 
synthesize an implementation from the axioms. This will 
permit testing at the level of the axiomatization without the 
trouble of implementation. Naturally such testing will be 
costly in terms of execution time, but experience indicates 
that only small cases are sufficient to turn up most errors. 

When testing is completed, a representation for the data 
type in terms of other data types is devised. Then imple¬ 
mentations of the operations in terms of the operations of 
the implementing data types are given. Once again testing 
is repeated, but this time DTVS will use the implementations 
as the semantics of the data type. Once we are satisfied, a 
proof of the correctness of the implementation is carried out 
semi-automatically by DTVS. 

Specification is a process we continue to explore. It is 
possible to build a complex software system by strictly ad¬ 
hering to algebraic axioms. By doing so one gains both the 
virtues of a formal approach and the testing and verification 
environment. This offers much more than an ad hoc strat¬ 
egy. Finally, this approach is not inconsistent with the even¬ 
tual use of a conventional programming language. 


PRACTICAL BENEFITS OF RESEARCH IN 

PROGRAMMING METHODOLOGY—Barbara Liskov 

In the past few years, considerable progress has been 
made in the area of programming methodology. Although all 
research in this area is interrelated, two main research di¬ 
rections can be distinguished. One direction is the study of 
software system structure, in particular study of desirable 
kinds of modules and module interconnections. The other 
direction is the study of the process of developing correct 
software having such desirable structure. 

Study of software system structure has been particularly 
effective within the framework of research on programming 
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languages, especially the languages CLU 1 and Alphard. 2 
This language work has succeeded in identifying new kinds 
of modules, and has provided precise rules governing the 
implementation and use of such modules. Each kind of mod¬ 
ule supports a kind of abstraction found to be useful in 
constructing software. The most important new kind of mod¬ 
ule is that supporting a data abstraction ; however, there are 
other kinds of modules, and in addition rules governing the 
interaction of modules. These latter rules constrain and re¬ 
duce the interconnections among modules. 

Earlier work in structured programming 3 and stepwise 
refinement 4 made evident the advantages of top down de¬ 
velopment of software. Recent work has elaborated the top 
down development process, taking into account the work on 
software system structure discussed above, and clarifying 
the way that top down development proceeds. The most 
important contribution has been the recognition of the role 
played by program specifications. A program specification 
is a description of the behavior of a module, the behavior 
that will be depended on by any user, and that must be 
provided by any implementation. At any stage of design, the 
goal is to identify lower level abstractions useful in imple¬ 
menting the current level. As these abstractions are identi¬ 
fied, their behavior is specified. The specification is given 
in advance of the implementation, and it provides a complete 
description of the interface of the module that will later 
implement the abstraction. The presence of the specification 
permits the question of how to implement the lower level 
modules to be deferred until a later stage of design. 

As programming methodology has become better under¬ 
stood, there has been increasing interest in defining formal 
methods to support it. In particular, there has been much 
recent research in formal specification techniques, 5,6 which 
permit the specifications discussed above to be expressed 
in a formal language, i.e., one with a well-defined and un¬ 
ambiguous syntax and semantics. The advantages of such 
formal specifications are twofold: they are more precise 
and concise than informal specifications, and therefore may 
serve better the role of interface descriptions described 
above, and, in addition, given formal specifications, a formal 
proof that a module’s implementation satisfies its specifi¬ 
cation is possible. However, formal specifications are more 
difficult to write than informal ones, and our current under¬ 
standing of specification and verification techniques is in¬ 
sufficient to permit all useful abstractions to be described. 

It is my belief that the present and near future construction 
of software systems can best be helped by popularizing the 
methodology. This can be done in a way that relies neither 
on a particular language, nor on the as yet incompletely 
understood specification and verification techniques. In¬ 
stead, the following two ideas must be made clear to pro¬ 
grammers: 

1. What constitutes good modularity. 

2. What constitutes good design practice. 

Rules about good modularity are best explained by devel¬ 
oping conventions, or better yet preprocessors, for existing 
languages in actual use; such conventions would permit a 


limited use of data abstractions and would prohibit current 
bad practices (such as non-local use of data). By expressing 
the rules in this way, they are explained in terms the pro¬ 
grammers can understand, and furthermore, a tool is pro¬ 
vided that helps in the development of a well-structured 
system. Good design practice then consists of top down 
decomposition into the kinds of modules that the program¬ 
mers already understand, with emphasis on the role of (in¬ 
formal) specifications in the process, and especially on the 
necessity of specifications being given in advance of imple¬ 
mentation. To aid programmers in understanding what spec¬ 
ifications are, it is helpful to establish a specification stand¬ 
ard which describes the kind of information that should be 
included in the specification, and gives a format for express¬ 
ing that information. However, it is too early to require that 
specifications be given in a formal language. Formal meth¬ 
ods in system design are not yet ready for practical use, but 
I believe use of the methods in an informal way can have 
considerable practical benefit. 
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BEYOND FACTORIAL—Donald I. Good 

Factorial is probably the single most thoroughly verified 
program ever. It, and other small examples such as stacks, 
typically are used in the literature to illustrate formal pro¬ 
gram verification methods, and this tends to create the 
impression that these small examples define the limits of 
effectiveness of formal verification. We must be careful not 
to paint an overly optimistic picture, but the view that formal 
verification methods are effective only on small examples, 
such as stacks and factorials, is equally overly pessimistic. 
Formal verification methods have been applied successfully 
to a variety of programs in the 1000-lines-of-code range and 
strong verification clauses are now being written into some 
software development contracts with the U.S. Government. 
Although there is much yet to be done, formal verification 
methods show considerable promise of becoming an effec¬ 
tive approach to building reliable software in actual practice. 

Basic research over the last ten years has done much to 
develop formal methods of program verification. The prob- 
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lem of actually verifying a 2000- to 5000-line program, how¬ 
ever, is much larger than just verification methods, and we 
must begin to focus some of our efforts on solving the 
“whole” verification problem if verification methods are to 
become effective in actual practice. The breadth and depth 
of the problem that is faced by a person charged with pro¬ 
ducing a substantial verified program is substantial. It en¬ 
compasses the following issues: verification methods, spec¬ 
ification methods, programming methods, specification 
language, verification language, programming language, 
specification tools, verification tools, programming tools, 
development strategies. 

Verification methods are the central issue, but are by no 
means the only issue. A verification demonstrates the con¬ 
sistency of a program and a specification. Therefore, attain¬ 
ing a verification requires using specification and program¬ 
ming methods that are both compatible with the verification 
methods. (By “programming methods” we mean the use of 
program constructs that are verifiable, as opposed to pro¬ 
gram “development strategies”.) We must know what kinds 
of things can and should be specified about various program 
constructs and we must know how to verify that the program 
construct meets the specification. We also must know how 
primitive verifiable program constructs can be put together 
to form large verifiable programs. Given the basic methods, 


precise languages for expressing the specifications, program, 
and verification are required. The specification and pro¬ 
gramming languages express the actual program and its de¬ 
sired properties, and the verifications are done in the veri¬ 
fication language which normally is some variation of the 
predicate calculus. The development of suitable languages 
involves all of the well-known problems of language design. 
Given these languages, effective tools are required to apply 
these languages to an actual problem. Without adequate 
tools, the languages and the methods they embody become 
effectively inaccessible. Finally, these tools must be used 
according to some reasonable strategy for step-wise program 
development. Just as independent compilation is of major 
importance in developing a large program, independent ver¬ 
ification is crucial even for a program of modest size; thus 
step-by-step verification must follow a sound program de¬ 
velopment strategy. Attaining a verified program of sub¬ 
stantial size requires effective solutions to all of these prob¬ 
lems and, just as importantly, it requires that these solutions 
be effectively integrated so that they can be used together 
in a systematic and coherent development of a verified pro¬ 
gram. In actual practice, an inadequate solution to any of 
these problems or an ineffective integration of individually 
effective solutions can completely defeat the verification of 
a 2000- to 5000-line program. 
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Automatic programming 


The term Automatic Programming is ill-defined. It means many different things 
to many people. It has covered everything from Fortran Compilers in the mid- 
50’s to futuristic visions of machines which completely program themselves to 
accomplish a given goal, or are even capable of generating those goals themselves. 

The three Automatic Programming sessions in this conference clearly cannot 
cover such a broad spectrum. Instead, they are focused on a set of small research 
projects each attempting to expand the computer’s involvement in the program¬ 
ming process. Surely, such continued expansion of computer involvement in the 
programming process is the central thrust of all the efforts in this field. 

The current state of this field can best be accessed by noting that programming 
is still largely a manual process. Significant progress has been made over the 
years, but it has largely been concentrated in the final stages of the programming 
process (such as language translators, test case generators, and debugging tools). 
What facilities exist which help us formulate and design systems, study tradeoffs 
between alternative designs, foresee the implications of design and implementa¬ 
tion decisions, understand the effects of modifications, predict system perform¬ 
ance, foresee system bottlenecks, or even prevent the introduction of bugs during 
implementation? 

Such tools basically do not exist. Instead, programmers and systems analysts 
deal with these issues in their own internal ways. These internal methods are not 
accessible for analysis by others, are error-prone, incomplete, and difficult to 
transfer from one individual to another. 

This deficiency has been recognized and responded to by the development of 
a set (actually several competing sets) of techniques for MANAGING the pro¬ 
gramming process. These techniques attempt to control the complexity of pro¬ 
gramming by advocating certain practices and styles while proscribing others, 
and by developing languages which encourage and/or enforce such direction. 
Certainly such endeavors have significantly improved our ability to control and 
manage these design and formulation processes occuring within our heads. 

Such manual techniques can only go so far. The regularized (structured) en¬ 
vironment that they have created must be augmented by various tools which 
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record the stepwise development of these processes as they occur within our 
heads, expose the decisions we are making, and enable us to see the implications 
of those decisions. 

The challenge for the field of Automatic Programming is to extend the pro¬ 
gramming environment by constructing such tools which expose this stepwise 
development process and enable each stage to be analyzed, as the basis for the 
next step, without impairing the formulators’ design flexibility. To accomplish 
this, there must exist a clear separation between a decision of what step to take 
next, which the formulator must control, and the carrying out of that decision, 
which the tool must perform to ensure that it is being accurately accomplished. 
In this way the integrity of the system being formulated is maintained as it moves 
from stage to stage. These tools thus ensure system integrity between formulation 
stages by automating the “clerical” operations which transform one stage into 
the next after a decision has been reached. Since a very large number of decisions 
must be made in any formulation process and some of them are very detailed, 
there may well exist some portion which the formulator relegates to the Automatic 
Programming tool. These Automatic Programming tools must also provide some 
analysis capabilities which enable the formulator to determine which decision to 
make next and how to make that decision. 

All of the prototype tools covered in these three Automatic Programming 
sessions match this paradigm. They differ in the portion of the formulation 
process they address, the analysis capabilities available at each stage, and the 
degree of control the formulator maintains over the decisions which get made. 

The first session concentrates on tools which address the process of specifying 
a program, while the second is composed of those which address the implemen¬ 
tation of such a specified system. These two sessions are not intended to present 
new research results (these have been reported in various specialized conferences 
and workshops), but rather, to bring together and highlight representative work 
in this field for the NCC audience. 

The final session, a panel, will attempt to access the extent and timeframe of 
the impact of such work within the computing community. 




Informality in program specifications 1 


by ROBERT BALZER, NEIL GOLDMAN and DAVID WILE 

Information Sciences Institute 
Marina del Rey, California 


A critical step in the development of a software system 
occurs when its goal-oriented requirements specification is 
transformed into a process-oriented form that specifies how 
the requirements are to be achieved. Only after this trans¬ 
formation has occurred can the feasibility of the system be 
analyzed and the consistency of the process specification 
with the requirements be verified. The key to this transfor¬ 
mation is expressing the process-oriented specification ab¬ 
stractly so that its functionality is completely determined 
while the class of possible implementations remains broad. 

We believe that such abstract process-oriented specifica¬ 
tions are the key to rationalizing the software development 
process. Such specifications are, in reality, programs written 
in a very high level abstract programming language. As such, 
they could provide an effective interface between the two 
major software concerns: functionality and efficiency. These 
concerns should be decoupled so that the functionality of a 
system can be addressed before its efficiency has been con¬ 
sidered. Once functionality has been accepted, it can be 
preserved while the system is optimized. Thus, since the 
abstract process-oriented specification is a program, its con¬ 
sistency with the requirements could be formally verified, 
informally argued, or tested by actually executing the spec¬ 
ification. Furthermore, the end user could be given hands- 
on experience exercising the specification to see if it be¬ 
haved as intended. Deviations and/or inconsistencies could 
be corrected in the specification before any implementation 
occurred. 

Once the system’s functionality has been accepted by the 
user, the efficiency of the system in meeting its performance 


requirements remains an issue. Such efficiency must be 
gained without altering the system’s accepted functionality. 
We have argued elsewhere 2 that a computer-based tool can 
be built which guarantees maintenance of functionality while 
a program is optimized without sacrificing the programmer’s 
ingenuity or initiative in determining how best to achieve 
efficiency. 

In this work we are concerned primarily with the proce¬ 
dure by which such process-oriented specifications are ob¬ 
tained and with computer-based tools for their construction. 
We will begin by determining some attributes of a suitable 
process-oriented specification language, then examine why 
specifications would still be difficult to write in such a lan¬ 
guage. We will argue that the key to overcoming these dif¬ 
ficulties is the careful introduction of informality based on 
partial, rather than complete, descriptions and the use of a 
computer-based tool which utilizes context extensively to 
complete these descriptions during the process of construct¬ 
ing a well formed specification. We will then present some 
results obtained by a prototype of such a computer-based 
tool on a few informal example specifications. Finally, we 
will discuss some of the techniques used by this prototype 
system. 

REFERENCES 

1. “Informality in Program Specifications,” IEEE Transactions on Software 
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The PSI program synthesis system, 1978—An abstract 

by CORDELL GREEN 

Stanford University 
Stanford, California 


This abstract will cover the current status of the PSI 
program synthesis system. PSI is a computer program that 
acquires high-level descriptions of programs and produces 
efficient implementations of these programs. PSI allows pro¬ 
gram specification dialogues using a mix of natural language, 
traces, examples, and a very high-level language. These 
specifications are used to construct a well-defined high-level 
program description of the desired program. This description 
is called a program model. The model is then refined into an 
efficient implementation of the program. PSI consists of 
several modules including an English parser-interpreter, 
trace and examples inference expert, domain expert, ex¬ 
plainer, dialogue moderator, program model builder, coder 
and efficiency expert. The class of programs synthesized are 
simple symbolic computation programs in either LISP or 
SAIL. 

The system design has been a group effort with the indi¬ 
viduals responsible for each module as follows: parser-in¬ 
terpreter, Jerrold M. Ginsparg; trace and example inference 
and domain experts, Jorge V. Phillips; moderator, Louis I. 
Steinberg; explainer, Richard P. Gabriel; model builder, 
Brian P. McCune; coder, David R. Barstow and Juan Lud¬ 
low; and efficiency expert, Elaine Kant. The interested 
reader should see the short summary paper 1 for a list of 
references on these components. 

PSI’s operation may be conveniently factored into an ac¬ 
quisition phase and a synthesis phase. The acquisition phase 
is concerned with acquiring the program specifications and 
producing a program model. The synthesis phase then finds 
an efficient implementation in the selected target language. 

An assumption used in the design of the acquisition phase 
of PSI is that there is no one best method for specifying all 
programs. So a mix of methods is allowed, ranging from a 
“conventional” very high-level language through English 
and traces. The English sentences are first parsed, then 
interpreted into fragments. The parser limits search by in¬ 
corporating considerable knowledge of English usage. The 
interpreter is more specific to automatic programming, using 
program description knowledge as well as knowledge of the 
last question asked and the current topic to facilitate the 
interpretation into fragments. 

Fragments form a loose description of program and data 
structure. The model builder must then apply knowledge of 
correct high level programs to convert the fragments into 
the model. The program model includes complete, consis¬ 


tent and executable (but slowly) high-level algorithm and 
information structures. The model builder processes frag¬ 
ments, checking for completeness and correctness, fills in 
detail, corrects minor inconsistencies, and adds cross-ref¬ 
erences. It also generalizes the program description, con¬ 
verting it into a form that allows the coder to look for good 
implementations. 

Another input specification method is partial traces mixed 
with input and output examples. Partial traces of states of 
internal and I/O variables allow the inductive inference of 
control structures. The trace and example inference module 
infers loose descriptions of programs in the form of frag¬ 
ments, rather than programs themselves. This technique 
allows domain support to disambiguate possible inferences, 
and also separates the issue of efficient implementation from 
the inference of the user’s intention. Application domain 
specific knowledge (e.g., knowledge about learning pro¬ 
grams) is concentrated in the domain expert, which supplies 
other acquisition modules with domain support for disam¬ 
biguation and program model completion. 

The moderator guides the dialogue with the user by se¬ 
lecting or repressing questions generated by various mod¬ 
ules. It attempts to keep PSI and the user in agreement on 
the current topic, provides a review-preview on a topic 
change, helps the user that gets lost, and allows initiative to 
shift between PSI and the user. The explainer module gen¬ 
erates questions about and descriptions of program models 
as they are acquired, in order to help verify that the inferred 
program description is the one desired. 

After the acquisition phase is complete, the synthesis 
phase begins. This phase may be viewed as a series of 
program refinements or as a heuristic search for an efficient 
program that satisfies the program model. The coder has a 
body of program synthesis rules that gradually transform the 
program model from abstract into more detailed constructs 
until it is in the target language. Both algorithm and data 
structures are refined interdependently. The coder deals pri¬ 
marily with the notions of sets and correspondences and can 
synthesize programs involving sequences, loops, simple 
input and output, linked lists, arrays, and hash tables. 

The refinement tree effectively forms a planning space 
that proposes only legal, but possibly inefficient, programs. 
The efficiency expert reduces the search of the tree by 
several heuristics and by estimating the time-space cost 
product of each proposed refinement and then using the 
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branch and bound method. It also factors the program into 
relatively independent parts so that all combinations of im¬ 
plementations are not considered. An analysis for bottle¬ 
necks can allocate synthesis effort to more critical parts of 
the program. 

In summary, we have formulated a framework for an 
automatic programming system and have a start on the kinds 
of programming knowledge that must be embedded therein. 
PSI is moderately successful in that a research prototype is 


currently running and has synthesized many different pro¬ 
grams including a graph traversal program, simple storage 
and retrieval programs and learning programs. 

REFERENCE 

1. Green, Cordell, “A Summary of the PSI Program Synthesis System,” 
Proceedings of the Fifth International Joint Conference on Artificial In¬ 
telligence, Cambridge, Massachusetts, August 1977, pages 380-381. 



Protosystem I—An automatic programming system 
prototype 


by GREGORY R. RUTH 

Massachusetts Institute of Technology 
Cambridge, Massachusetts 


INTRODUCTION 

Over the years people have come to understand certain 
functions of the programming process well enough to auto¬ 
mate them—that is to replace those functions by programs. 
The most notable results were assemblers, compilers, and 
operating systems. Our knowledge and understanding of 
programming is once again reaching a level where a signif¬ 
icant advance in automation is both necessary and possible. 
In particular, it is now possible (for certain simple applica¬ 
tion domains) to create a system that will take as input a 
specification for a user’s application and will automatically 
design and code the desired data processing system. To 
demonstrate feasibility and gain insight into the issues and 
technology involved in creating such a system, a prototype 
automatic programming system (Protosystem I) for gener¬ 
ating business data processing systems is currently being 
developed at MIT. 


A MODEL OF THE PROGRAM WRITING PROCESS 

The data processing system writing process may be con¬ 
ceived as a sequence of phases leading from the conception 
of a system to its implementation as executable machine 
code. A useful and plausible model for this sequence of 
phases is: 

Phase 1: Problem Definition—The system specification is 
expressed in domain dependent terms in English that is 
understandable by the program developers. 

Phase 2: General System Analysis and Design—The prob¬ 
lem is reformulated in standard data processing terms and 
expressed as an instance of a known solvable problem 
class (in our case a subset of the class of all batch oriented 
dps’s). Domain dependent policy and procedures are 
worked out in detail at this stage. 

Phase 3: System Implementation—The system—the ac¬ 
tual procedural steps and data representations and orga¬ 
nizations—is constructed by intelligent selection from and 
adaptation of a number of standard implementation pos¬ 
sibilities. 


Phase 4: Code Generation—The design specifications are 
implemented in a high-level language (e.g., PL/I, COBOL) 
in a fairly straightforward, but not totally mechanical, 
way. 

Phase 5: Compilation and Loading—A form is produced 
that can be “understood” and executed by the target 
computer. 

These phases progress from a general notion of what is to 
be done by the desired system toward a detailed specifica¬ 
tion of how it can be accomplished. They also represent the 
classes of design and implementation problems involved in 
program writing, progressing from the most global and gen¬ 
eral considerations toward the most local and detailed is¬ 
sues. 

Protosystem I seeks to automate the program writing 
process by automating and tying together the phases de¬ 
scribed in the model given above. That is, Protosystem I is 
designed in such a way that there are explicit parts or stages 
corresponding to each of the model phases. Each such stage 
embodies the knowledge and expertise of the human agent(s) 
for the corresponding phase, so that, given the same or 
similar input, it can intelligently produce comparable cor¬ 
responding results. 

The products of each stage should be sufficiently general 
and malleable so that further stages can have the maximum 
freedom in making their design contributions in the most 
effective and efficient ways. Consequently, we have chosen 
in Protosystem I to make the product of each stage a de¬ 
scriptive representation of the dps in terms of concepts and 
considerations appropriate for the next stage of develop¬ 
ment. In this way the programming process is conceived as 
the development of a succession of ever more precise system 
descriptions until, ultimately, a level is reached where every 
detail has been decided and the result is an executable com¬ 
puter program. 

EFFICIENCY ENHANCEMENT IN SYSTEM 
DEVELOPMENT 

To produce a credible and practical result an automatic 
programming system must perform a reasonable degree of 
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optimization. Current formal optimization methods pertain 
mainly to the compilation level; when the entire program 
development process is automated, new, additional types of 
optimization will have to be included. 

The various possible types of optimizations fall quite nat¬ 
urally into categories that correspond to the program writing 
levels in our model. For instance, the combination of com¬ 
putations (for the sake of I/O efficiency) is something that 
should be considered during Stage 3 (system implementa¬ 
tion) where the data and computational interrelationships 
among conceptual processing units are most evident. Prob¬ 
lems involving efficient loop construction should be handled 
in a Stage 4. 


THE DEVELOPMENT OF PROTOSYSTEM I 

The Protosystem I effort began in 1971. Because of the 
difference in the natures of the technologies involved the 
work was divided into two parallel efforts: (1) a top-part-of- 
the-system effort (Phases 1 and 2), essentially of an Artificial 
Intelligence nature, concerned with natural language com¬ 
prehension, model formation and problem solving, (2) a bot- 
tom-part-of-the-system-effort (Phases 3 and 4) addressing 
the problems of implementation and optimization of a pro¬ 
gram given an abstract relational specification (ultimately to 
be passed down from the top part of Protosystem I). The 
bottom part of Protosystem I has been completely imple¬ 
mented in the MACLISP language and is operational on the 
MIT LCS PDP-10. Research and development on the top 
part, being considerably more ambitious and novel, is not 
expected to cross the threshold of practical applicability for 
another five years, and so will not be discussed further in 
this article. 

A structural diagram of the bottom part of Protosystem I 
is shown in Figure 1. The following sections give an expla¬ 
nation of its workings. 


THE PROTOSYSTEM I DATA PROCESSING SYSTEM 
MODEL AND THE SYSTEM SPECIFICATION 
LANGUAGE 

Protosystem I handles a restricted but significant subset 
of all data processing applications: I/O intensive batch ori¬ 
ented systems. Such systems involve a sequence of runs or 
job steps that are to be performed at specified times. They 
are assumed to involve significant I/O activity due to repet¬ 
itive processing of keyed records from large files of data. 
Systems such as inventory control, payroll, and employee 
or student record keeping systems are of this type. 

A simple example of such a dps is a software system to 
perform the inventory and warehousing activities in the fol¬ 
lowing case: 

The A&T Supermarket chain consists of 500 stores 
served by a centrally located warehouse. There are 4000 
items, supplied by the warehouse, that these stores can 
carry. 



Data flow- 

Calls- 

Transfer of control 

Figure 1—Protosystem I—Structure of the bottom part 

Every day the warehouse receives shipments from sup¬ 
pliers and updates its inventory level records accordingly. 

It also receives orders from the stores for various quan¬ 
tities of items. If for a particular item there is sufficient 
stock to fill all of the orders for that item, the warehouse 
simply fills the orders as made; but if there is insufficient 
stock it ships partial orders proportional to fraction of the 
total quantity ordered that is on hand. 

Inventory records are adjusted to reflect the decrease 
in levels. 

Finally, a daily check is made on the inventory levels 
of all items. If the level of an item is lower than 100 the 
warehouse orders 1000 more units of that item from the 
appropriate supplier. 

In order for the bottom part of Protosystem I to implement 
such a data processing system application the basic aggre¬ 
gate data entities and their interrelationships must be deter¬ 
mined. Consider the inventory updating activity of the sec¬ 
ond paragraph. There are three aggregate data entities 
involved: (1) the set of quantities received from suppliers, 
(2) the set of closing inventory levels for the previous day, 
and (3) the set of the updated inventory levels to be used 
for filling store orders. Such collections of similar data that 
are processed in a similar way are termed data sets. In 
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Protosystem I a data set is assumed to consist of fixed 
format records (e.g., one for the level of each inventory 
item). Associated with each record is a data item (e.g., the 
level of an inventory item) and keys. The key values of a 
record uniquely distinguish it (e.g., the inventory data set 
can be keyed by item since there is only one level [record] 
per item) and so can be used to select it.* If we call these 
data sets shipments-received, final-inventory and be¬ 
ginning-inventory, the relationship between beginning- 
inventory and the others may be described as: 

For every item: 

the beginning inventory level of that item 
(i.e. the value of the data item for the record in begin¬ 
ning-inventory for that item) 

is the closing inventory level of that item from the 
previous day 

(i.e. the value of the data item in the record of final- 
inventory for the same item) 
plus the quantity of that item received 
(i.e. the value of the data item in the record of ship¬ 
ments-received for the item in question), 
if any. 

This relationship is expressed more succinctly in SSL (the 
System Specification Language):** 

beginning-inventory is final-inventory(1 day 
ago) + shipments-received 

The repetitive application of an operation to the members 
of a data set or sets such as this is termed a computation. 
The order of applications of the operation to the records of 
its input data sets by a computation is assumed to be un¬ 
important to the user; in fact, he may think of them as being 
performed in parallel. However, every computation does, in 
fact, process its input serially, according to a particular 
ordering (chosen by Protosystem I) on their keys. Compu¬ 
tations typically match records from different data sets by 
their keys (as above) and operate on the matching records 
to produce a corresponding output record. A computation 
may also group the members of a data set by common keys 
and operate on each group to produce a single corresponding 
output. Returning to our example, note that item orders can 
come from different sources (stores), so that both the item 
and the source of an order are needed (as keys) to distinguish 
it. To form the total of all orders for each item, a compu¬ 
tation must group the orders by item and sum over the order 
amounts in each group. In SSL this would be expressed as: 

TOTAL-ORDERS FOR EACH ITEM IS THE SUM OF THE 
QUANTITY-ORDERED-BY-STORE 

Figure 2 shows the structure of the A&T inventory and 
warehousing data processing system in terms of computa¬ 
tions (boxed) and data sets (unboxed). The complete SSL 


* Thus, a data set is essentially the same as a Codd relation and its keys are 
what Codd calls primary keys. 

** Implicit in this statement is that the addition operation is performed for 
each item and that if one of the operands is missing (e.g., if no chicken noodle 
soup was received today) it is treated as having a zero value. 


description of A&T dps is given in Figure 3. Note that in 
addition to the relational statements a list of data sets must 
be included to indicate the keys by which they are accessed. 

THE TRANSLATOR AND THE DATA SET 
LANGUAGE 

For the dps’s which Protosystem I proposes to treat the 
calculations themselves are easily dealt with; the structuring 
and manipulation of the masses of data involved forms the 
major part of the Stage 3 implementation activity. Conse¬ 
quently, the development process at Stage 3 is data set 
oriented. Therefore, to facilitate the design process the SSL 
dps description is first analyzed from this point of view and 
re-expressed in a more appropriate medium, DSL (the Data 
Set Language). This reformulation is performed by the 
Translator module. 

The determination of dps characteristics that can aid in 
the development of the dps design is made with the aid of 
the Structural Analyzer and included in the Translator’s 
output description. This output is called the UDSL (Uncon¬ 
strained Data Set Language) description, because most de¬ 
sign details remain unbound (undecided) in it. As such it 
forms the skeleton of the dps description ultimately to be 
produced by Stage 3. 

One useful piece of information determined by the Struc¬ 
tural Analyzer is the set of driving data set candidates for 
each computation. A driving data set is an input data set 
that is guaranteed to have a data item for every tuple of key 
values for which the computation can produce an output. 
The computation, then, instead of having to loop over all 
possible combinations of values for the keys of the inputs, 
can be driven by the driving data set in that it only has to 
consider those key value combinations for which the driving 
data set contains records. 

Another type of information the Structural Analyzer de¬ 
termines is directly related to our desire to specify data set 
organizations and orders and computation accessing meth¬ 
ods and orders in such a way as to minimize the cost of 
operating the dps. Because a dps typically involves the re¬ 
petitive application of simple calculations to large quantities 
of data we make the first-order approximation that the cost 
of operation is due entirely to data accessing (reading and 
writing). Our design, therefore, focuses on minimizing the 
total number of I/O events. 

Accordingly, the Structural Analyzer also determines 
predicates that are the conditions under which a data item 
will be generated and under which a data item will be used 
by a computation. For example, a store will be shipped an 
item if (it is true that) that store ordered that item and there 
was sufficient inventory to fill the order; the order allocation 
step will use the inventory level for a particular item if some 
store ordered it. These predicates, together with basic in¬ 
formation concerning the sizes of data sets in the dps, are 
used by the Question Answerer to determine the average 
and maximum sizes of files (proposed by the Optimizing 
Designer) and the average number of a file’s records a com¬ 
putation will access. 
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yesterday’s supplier quantities of each 

final shipments item ordered by stores 



final inventory 




reorder calculation 




reorder amounts 

Figure 2—A&T inventory and warehousing system 


THE DESIGN CRITERION AND THE JOB COST 
ESTIMATOR 


The design criterion for Protosystem I is the minimization 
of the dollars and cents cost of running the final dps program 
on the target machine/operating system configuration. Be¬ 
cause the dps’s are assumed to be I/O intensive, as a first 
approximation, this can be equated with access minimiza¬ 
tion. An access in this sense is defined as the reading or 
writing of a single secondary storage block, which corre¬ 


sponds to a single operating system I/O event. In Protosys¬ 
tem I, for a particular data set a block consists of a fixed 
number of records. 

With this approximation the relative costs of alternative 
dps design configurations can often be assessed without 
knowledge of the particular target configuration. But some¬ 
times actual cost estimates, provided by the Job Cost Esti¬ 
mator, are necessary. This module must thus contain knowl¬ 
edge of the charging scheme and operating characteristics 
of the target configuration (in our case the OS/360 configu¬ 
ration). Optimization with respect to a different configura- 
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DATA DIVISION 


FILE SHIPMENTS-RECEIVED 
KEY IS ITEM 
GENERATED EVERY DAY 

FILE BEG INNING-INVENTORY 
KEY IS ITEM 
GENERATED EVERY DAY 

FILE TOTAL-ITEM-ORDERS 
KEY IS ITEM 
GENERATED EVERY DAY 

FILE QUANTITY-SHIPPED-TO-STORE 
KEYS ARE ITEM, STORE 
GENERATED EVERY DAY 


FILE QUANTITY-ORDERED-BY-STORE 
KEYS ARE ITEM, STORE 
GENERATED EVERY DAY 

FILE TOTAL-SHIPPED 
KEY IS ITEM 
GENERATED EVERY DAY 

FILE FINAL-INVENTORY 
KEY IS ITEM 
GENERATED EVERY DAY 

FILE REORDER-AMOUNT 
KEY IS ITEM 
GENERATED EVERY DAY 


COMPUTATION DIVISION 

BEG INNING-INVENTORY IS F INAL-INVENTORY( 1 DAY AGO) + SHIPMENTS-RECEIVED 
TOTAL-ITEM-ORDERS IS SUM OF QUANTITY-ORDERED-BY-STORE FOR EACH ITEM 
QUANTITY-SHIPPED-TO-STORE IS 


QUANTITY-ORDERED-BY-STORE IF BEG INNING-INVENTORY IS 6REATER 

THAN TOTAL-ITEM-ORDERS 


QUANTITY-ORDERED-BY-STORE 
* (BEGINNING-INVENTORY / TOTAL-ITEM-ORDERS) 

IF BEGINNING INVENTORY IS NOT 
GREATER THAN TOTAL-ITEM-ORDERS 


TOTAL-SHIPPED IS SUM OF QUANTITY-SHIPPED-TO-STORE FOR EACH ITEM 
FINAL-INVENTORY IS BEGINNING-INVENTORY - TOTAL-SHIPPED 

REORDER-AMOUNT IS 1000 IF F INAL-INVENTORY IS LESS THAN 100 

Figure 3—SSL relational description for the A&T data processing system 


tion and/or charging scheme would require the substitution 
of a new appropriately tailored module. 

THE QUESTION ANSWERER 

The function of the Question Answerer is to supply an¬ 
swers to questions from the Optimizing Designer about the 


average sizes (in records) of abstract aggregate data entities. 
Two examples of such data aggregates are a file and the 
collection of records in a file that are accessed by a particular 
computation. Each “question” sent to the Question An¬ 
swerer is in the form of a predicate describing the conditions 
under which a record will be in the data aggregate in ques¬ 
tion. 
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The Question Answerer maintains a data base of all of the 
event probability and size information given by the user. 
When asked a question it attempts to find the associated 
size or probability directly. Failing this, it will try to calcu¬ 
late the probability of the event in question happening from 
those of its sub-events and its knowledge of event inde¬ 
pendence and correlation within the dps. If the information 
on hand is insufficient to answer the question, the Question 
Answerer obtains enough additional information from the 
user (through a flexible line of questioning) to do so. 


THE OPTIMIZING DESIGNER 

The Optimizing Designer is the heart of Stage 3; all of the 
other modules in this stage exist merely to serve it. When 
the translation from SSL to UDSL has been completed, 
control passes to the Optimizing Designer. This module is 
responsible for constructing job steps to implement com¬ 
putations and files to implement data sets. In particular its 
job is to: 

1. design each keyed file—in particular its 

a. contents (information contained) 

b. OS/360 organization (consecutive, index sequen¬ 
tial, or regional(2)) 

c. storage device 

d. associated sort ordering (by key values) 

e. blocking factor (number of records per block) 

2. design each job step of the dps—namely 

a. which computations it includes 

b. its accessing method (sequential, random, core 
table) 

c. its driving data set(s) 

d. the order (by key values) in which it processes the 
records of its input data sets 

3. determine whether sorts are necessary and where they 
should be performed 

4. determine the sequence of the job steps 

The Optimizing Designer performs dynamic analysis (anal¬ 
ysis of the operating behavior) on the dps to propose and 
evaluate alternative design configurations. Occasionally, 
static analysis (analysis of system structure and interrela¬ 
tionships) of such tentative configurations is also necessary, 
and this is obtained through calls to the Structural Analyzer. 
When additional information is needed to make evaluations 
and decisions the Question Answerer and the Job Cost Es¬ 
timator are called. 

All design decisions are made in an effort to minimize the 
total number of accesses that must be performed in the 
execution of the dps. There are three major techniques that 
the Optimizing Designer uses toward this end: 

1. Designing files and job steps to take advantage of 
blocking —Accesses can be reduced if files are given 
blocking factors greater than one and if processing and 


file organizations are designed in such a way that the 
records of each block can be used consecutively. 

2. Aggregating data sets —If two or more data sets that 
are accessed by the same computation are combined 
into one file (whose records have multiple data items) 
and processing is arranged so that a single record of 
the aggregate can be accessed where more than one 
record from each of the otherwise unaggregated files 
would have been accessed, accesses can be saved. 

3. Aggregating computations —When two or more com¬ 
putations access the same data set and the orders in 
which they process the records of that data set are the 
same, it may be advantageous to combine them into a 
single job step. Then each record of the shared data 
set can be accessed once for all, rather than once for 
each computation. 

These access minimizations techniques require that the 
key order of processing agree in a special way with the 
organization of the data being processed. Because different 
computations accessing the same data set may have different 
preferences for its organization, optimization of the type 
performed is necessarily a problem in global compromise. 

The straightforward solution of evaluating the cost of 
every possible combination of assignments of sort order, 
device, organization, and access method for data sets and 
computations in every possible aggregation configuration to 
determine the least expensive is ruled out by the sheer 
combinatorics involved. Even with mathematical and special 
purpose tricks it would be impossibly slow. 

To make optimization tractable a heuristic approach must 
be taken. First different kinds of decisions (e.g., choice of 
driving data sets, which objects to aggregate) must be de¬ 
coupled wherever possible. Further decoupling must be ju¬ 
diciously introduced where it is not strictly possible, for the 
sake of additional simplicity. Such forced decoupling does 
not mean, though, that decisions that are in fact coupled are 
treated as if they were independent. The decoupled deci¬ 
sions are still made with a certain awareness of their effects 
on other decisions. Finally, as a first order approximation, 
the optimizer does what is reasonable locally, and then ad¬ 
justs somewhat for global realities. While we make no claim 
that this approach will lead to the true optimum, it does 
produce good and usually near-optimal solutions for real and 
honest problems. 


STAGE 4: CODE GENERATION 

Stage 4 of Protosystem I consists of the PL/I and JCL 
Generator modules. The PL/I Generator takes the fully spec¬ 
ified output of Stage 3 (the CDSL or Constrained Data Set 
Language description) as input and produces PL/I code for 
each job step. This involves the determination and arrange¬ 
ment of PL/I I/O specifics, the construction of the data 
processing loops, and the programming of the necessary 
calculations. The JCL Generator then writes IBM OS/360 
JCL and ASP instructions for the I/O, administration and 
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scheduling of the compilation and execution of the dps job 
and job steps. 


CONCLUSION 

A model of the data processing system implementation proc¬ 
ess has been presented and a blue-print, based on that model, 
for automating the entire process has been developed. Pro¬ 
tosystem I is a project to exhibit the feasibility of these 
ideas. Already, two of the four heretofore manual phases of 
the software writing process have been automated and are 
capable of producing acceptable implementations. The au¬ 
tomation of the remaining two phases should easily fall 
within the realm of presently developing technologies within 
the next decade. 
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INTRODUCTION 

Program synthesis is the automatic construction of programs 
to meet given specifications. These specifications constitute 
a high-level description of the desired program which ex¬ 
presses the purpose of the program, without indicating the 
method by which that purpose is to be achieved. 

The specifications are expressed in terms of many con¬ 
structs which are endemic to the particular subject domain 
of the desired program (e.g., numbers, sets, lists). Because 
these constructs are only intended to describe the purpose 
of the program and need not be computed, they can be of 
a much higher level than the constructs of any programming 
language (e.g., they can include logical quantifiers, set con¬ 
structors, and other noncomputable operations). The spec¬ 
ification language can correspond closely with the concepts 
a programmer actually uses in thinking about the problem. 

The techniques we are developing are independent of the 
choice of a target programming language. The particular 
language we use in our examples and in our experimental 
system is a simple LISP-like language containing only basic 
numerical and list-processing operations, conditional 
expressions, and recursion. In considering the formation of 
programs with side effects, we extend the language to in¬ 
clude assignments to variables, array elements, and other 
data-structure components. 

Our basic approach is to transform the specifications re¬ 
peatedly according to certain rules; each rule replaces one 
segment of a program description by another, equivalent, 
segment. The process continues until a description is ob¬ 
tained that is entirely in terms of the primitive constructs of 
the target language; this description is the desired program. 


* This research was supported in part by the National Science Foundation 
under Grants DCR72-03737 A01 and MCS76-83655, by the Office of Naval 
Research under Contracts N00014-76-C-0687 and N00014-75-C-0816, by the 
Advanced Research Projects Agency of the Department of Defense under 
Contract MDA903-76-C-0206, and a grant from the United States-Israel Bi¬ 
national Science Foundation. 


The entire sequence of descriptions leading from the speci¬ 
fications to the final program is called a program derivation. 

The transformation rules are guided by certain strategic 
controls, which ensure that they are applied only at the 
appropriate time. Many of the transformation rules represent 
knowledge about the program’s subject domain; some ex¬ 
plicate the meaning of the constructs of the specification 
and target languages; a few rules correspond to basic pro¬ 
gramming principles, which are independent of the particular 
subject domain or programming language. 

The programs constructed by these techniques are guar¬ 
anteed to be correct and to terminate, and require no sep¬ 
arate verification phase. Up to now, we have not been con¬ 
cerned with the efficiency of the target program. 

The techniques we develop are tested in the implemen¬ 
tation of an experimental system called DEDALUS (DE¬ 
Ductive ALgorithm Ur-Synthesizer). This system is imple¬ 
mented in the QLISP programming language, an extension 
of INTERLISP that includes pattern-matching and back¬ 
tracking facilities. 

The identification of the basic programming principles, 
their codification as transformation rules, and their imple¬ 
mentation in our experimental system have constituted a 
major component of our effort. Some of the principles we 
have identified so far are: 

• Conditional formation —This principle causes a case 
analysis to be introduced into the derivation, yielding 
a conditional expression (or test) in the ultimate pro¬ 
gram. 

• Recursion formation —This principle introduces a re¬ 
cursive call into the ultimate program by observing 
when a subgoal to be achieved is actually an instance 
of the desired top-level goal. 

• Well-founded ordering —The termination of the recur¬ 
sive programs formed by the above technique is ensured 
by constructing a well-founded ordering with the prop¬ 
erty that the arguments of the program’s recursive calls 
are all strictly less than the program’s inputs. 
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• Procedure formation —A subsidiary procedure is 
formed when a subgoal is found to be an instance, not 
of the top-level goal, but of a previously generated 
subgoal. 

• Generalization —A generalized procedure is formed 
when two subgoals are found to be an instance of a 
third expression, which is somewhat more general than 
both. 

• Simultaneous goals —In constructing a program to 
achieve two or more goals simultaneously, we first con¬ 
struct a program to achieve one goal, then modify that 
program to achieve the others as well, while protecting 
the condition that was already achieved. 

Some of these principles are fairly well understood and have 
been incorporated into the DEDALUS system; others re¬ 
quire further study. In the next sections we examine in some 
more detail our basic program-synthesis techniques, indi¬ 
cating which areas have already been implemented. Our 
treatment of these techniques will be extremely brief; further 
discussion of the same topics, at a more leisurely pace, 
appears in the authors’ Stanford University-SRI Interna¬ 
tional report, “Synthesis: Dream subprograms ” (November 
1977). 


GENERAL FRAMEWORK 

Specifications 

In designing the specification language, we have incor¬ 
porated many constructs (e.g., the set constructor and the 
logical quantifiers) that facilitate the description of a pro¬ 
gram but that may not be included in the target programming 
language. We present below examples of specifications for 
simple programs using some of these high-level constructs. 

A program lessall(x l), to test if a number x is less than 
every element of a list / of numbers, is specified as follows: 

lessall(x l )<f compute x <all(l) 
where x is a number and 

/ is a list of numbers. 

In general, the specification construct P(all(l)) denotes that 
the property P holds for every element of the list /. 

The specification for a program to compute the greatest 
common divisor gcd (x y) of two nonnegative integers * and 
y is 

gcd(x compute max{z :z\x and z\y} 

where x and y are nonnegative integers and 
or yi=0. 

The set constructor {«: P(u)} denotes the set of all elements 
u satisfying the property P. 

The all construct ( P(all(l )) and the set constructor 
{u: P(u)} are specification constructs that are nonprimitive 
(i.e., they are not in the target programming language). The 
synthesis task is to transform a description of the desired 
program, such as the specifications presented above, into 


an equivalent description that employs only primitive con¬ 
structs of the target language. 

The specification language is not fixed: as we consider 
new subject domains, we introduce new specification con¬ 
structs accordingly. 

Transformation rules 

We use the notation 
t t' if P 

to denote that a subexpression of form t may be replaced 
by the corresponding expression t', provided that the con¬ 
dition P is true. 

For example, the rule 

Q and true => Q 

denotes the basic logical principle that an expression of form 
“Q and true” may be replaced by Q. This rule has no 
conditions; it can always be applied. 

The rule 

P(all (/))=> PQiead ( l))andP(all(tail (/))) if not empty (/) 

expresses the fact that a property P holds for every element 
of a nonempty list / if it holds for the first element head(l ) 
and for every element of the list tail(l ) of the other elements. 
This rule imposes the condition that the list l be nonempty. 

In the DEDALUS system, transformation rules are rep¬ 
resented as programs in the QLISP language. The full ex¬ 
pressive power of the programming language may be brought 
to bear in representing each rule. 

Derivation trees 

In developing a program whose specifications are 

/(*)<= compute P(x) 
where Q(x), 

we establish the output description as a goal to be achieved, 
viz., 

Goal: compute P(x). 

Subgoals are derived from this goal by application of the 
relevant transformation rules. For example, in deriving the 
gcd program, we form the top-level goal 

Goal 1: compute max{z\ z\xand z\y} 

By applying a transformation rule 

u\vand u\w u\vand u\w-v 

we obtain the subgoal 

Goal 2: compute max{z : z\x and z\y-x}. 

If a transformation rule imposes a condition P, which 
must be true if the rule is to be applied, a subgoal of the 
form 

Goal: prove P 
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must be achieved before the rule can be applied. For ex¬ 
ample, in developing the program lessall(x /) to test if a 
number x is less than every element of a list / of numbers, 
we have the top-level goal 

compute x<all (/), 

which is obtained directly from the specifications; in at¬ 
tempting to apply the rule 

P{all (/))=> true if empty (l ) 
to this goal, we derive the subgoal 

Goal: prove empty (/). 

From each subgoal that is derived, we can generate further 
subgoals by the application of more transformation rules. 
We thus construct a tree of goals and subgoals, which we 
will call a program derivation tree. 

A subgoal compute S is already achieved if S consists 
entirely of primitive constructs of the target language. A 
subgoal prove P is achieved if P is the logical constant true. 
Such goals are terminal nodes of the derivation tree. 

Let us now discuss some of the basic programming prin¬ 
ciples used in developing the derivation tree. 


BASIC PROGRAMMING PRINCIPLES 
Conditional formation 

Many of the transformation rules impose conditions (e.g., 
/ is nonempty, x is nonnegative) that must be satisfied for 
the rule to be applied. Suppose that in attempting to apply 
a particular rule, we fail to prove or disprove a condition P, 
where P is expressed entirely in terms of the primitive con¬ 
structs of the programming language; in such a situation, the 
conditional-formation rule is invoked. This rule allows us to 
introduce a case analysis, and consider separately the case 
in which P is true and P is false. Suppose we succeed in 
constructing a program segment 5 X that solves our problem 
under the assumption that P is true, and another program 
segment S 2 that solves the problem under the assumption 
thatP is false. Then the conditional-formation principle puts 
these two program segments together into a conditional 
expression 

if P then 5 X else S 2 , 

which solves our problem regardless of whether P is true or 
false. 

If we happen to generate the program segment S 2 , say, 
without using the case assumption that P is false, then S 2 
solves our problem regardless of whether or not P is true. 
In this case, no conditional expression is formed, and the 
program constructed is simply S 2 . Thus, conditional expres¬ 
sions are generated only for truly relevant conditions. 

The conditional-formation rule is among the best-under¬ 
stood of our basic programming principles and was among 
the first to be incorporated into the DEDALUS implemen¬ 
tation. Other program-synthesis systems that form condi¬ 


tional expressions by case analysis have been implemented 
by Buchanan 1 and Luckham and Warren. 2 


Recursion formation 

Suppose, in constructing a program whose specifications 
are 

/UK^compute P(x) 
where Q(x) 

we encounter a subgoal 

compute P(t) 

which is an instance of our output specification, compute 
P(x). Because the program f(x) is intended to compute P(x) 
for any x satisfying its input specification Q(x), the recur¬ 
sion-formation rule proposes achieving the subgoal by com¬ 
puting P(t) with a recursive call fit). For this step to be 
valid, it must ensure that the input condition Q(t ) holds 
when the proposed recursive call is executed. To ensure 
that the new recursive call will not cause the program to 
loop indefinitely, the rule must also establish a termination 
condition, showing that the argument t is strictly less than 
the input x in some well-founded ordering. (A well-founded 
ordering is one in which no infinite strictly decreasing se¬ 
quences can exist.) This condition precludes the possibility 
that an infinite sequence of recursive calls might occur dur¬ 
ing the execution of the program. 

For example, the DEDALUS system derived the program 
lessall (x /), which tests whether a given number x is less than 
every element of a given list / of numbers. The specifications 
for this program are 

lessall (x /)<=compute x<all (/) 
where x is a number and 

l is a list of numbers. 

In deriving this program, we develop a subgoal 
compute x<all{tail (/)) 

in the case that / is nonempty. This subgoal is an instance 
of our output specification, with the input l replaced by 
tail(l); therefore, the recursion-formation principle proposes 
that we achieve the subgoal by introducing a recursive call 
lessall(x tail(l)). To ensure that this step is valid, the rule 
establishes an input condition, that 

x is a number and 

taifil ) is a list of numbers, 

and a termination-condition that the argument pair (x tail (/)) 
is less than the input pair (x /) in a well-founded ordering. 
This termination condition holds because tail (l ) is a proper 
sublist of /. 

The final program we obtain is 

lessall (x empty {l) 
then true 

else x<head (/) and lessall (x tail (/)). 
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The recursion-formation principle is well-understood and 
is included in the DEDALUS implementation. The principle 
is the same as the “folding rule” of the Burstall and Dar¬ 
lington 3 program transformation system. 

Procedure formation 

Suppose in developing a program whose specifications are 
of the form 

/(x)<=compute P(x) 

where Q(x) 

we encounter a subgoal 
Goal B: compute R(t), 

which is an instance, not of the output specification compute 
P(x), but of some previously generated subgoal 

Goal A: compute R(x). 

The procedure-formation principle proposes that we intro¬ 
duce a new procedure g(x) whose output specification is 

g(xX=compute R(x). 


where 

carthead (s t)<^if empty (t) 
then {} 

else {(head (5) head (/))} U 
carthead (s tail ( t )). 

The basic procedure-formation principle has been imple¬ 
mented, and the DEDALUS system can carry out this and 
other such derivations. However, our method for proving 
the termination of ordinary recursive calls does not always 
extend to the multiple-procedure case. 

Generalization 

Suppose in deriving a program we obtain two subgoals 
Goal A: compute R(a(x)) 
and 

Goal B: compute R(b(x)), 

neither of which is an instance of the other but both of which 
are instances of the more general expression 

compute R(y). 


In this way, we can achieve both Goals A and B by calls 
g(x) and g(t) to a single procedure. In the case that Goal B 
has been derived from Goal A, the call to g(t) will be a 
recursive call; otherwise, both calls will be simple procedure 
calls. 

For example, in constructing a program cart(s t ) to com¬ 
pute the Cartesian product s and t), of two sets, we are 
given the specification 

cart(s recompute {(x y): xes and yet} 
where s and t are finite sets. 

In deriving the program, we obtain a subgoal 

Goal A: compute {(x y):x=head(s) and ye/} 

in the case that s is nonempty. Developing Goal A further, 
we derive the subgoal 

Goal B: compute {(x y)\x=head(s) and ye tail(t)} 

in the case that t is nonempty. Goal B is an instance of Goal 
A; therefore, the procedure-formation rule proposes intro¬ 
ducing a new procedure carthead (5 t) whose output speci¬ 
fication is 

carthead (s recompute {(x y):x=head(s) and yet} 

so that we can achieve Goal A with a procedure call cart¬ 
head (s t) and Goal B with a (recursive) call carthead (s 
tail(t)). 

The carthead procedure is constructed by the techniques 
we have already introduced. The final system of programs 
we obtain is 

cart (5 t'fcif empty (s) 
then {} 

else carthead (s t ) U 
cart(tail(s) t ), 


Then the extended procedure-formation rule proposes that 
we introduce a new procedure, whose output specification 
is 


g(y)<=compute R(y), 

so that we will be able to satisfy Goal A by a procedure call 
g(a(x)) and Goal B by a procedure call g(b(x)). 

For example, in constructing a program reverse (/) to 
reverse a list /, we derive two subgoals 

Goal A: compute append(reverse(tail(l)) 
cons(head(l) 
nil)) 


and 


Goal B: compute append(reverse(tail(tail (/))) 
cons(head(tail (/)) 

cons(head (/) nil))). 

Each of these goals is an instance of the more general 
expression 

compute append(reverse(tail (/)) 
cons(head(l) 
m)); 

therefore, the extended procedure-formation rule proposes 
introducing a new procedure reversegen (l m), whose output 
specification is 

reversegen (l ra)<=compute append(reverse(tail (/)) 

cons(head (l) 

m)). 

This procedure reverses a nonempty list / and appends 
the result to m. Although the procedure solves a more gen¬ 
eral problem than the reverse program we actually require. 
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it turns out that the reversegen procedure is actually easier 
to construct. The final system of programs we obtain is 

reverse empty (l) 

then nil 

else reversegen (l nil ) 

where 

reversegen {l m)<=if empty (tail (l )) 

then cons(head(l) m) 
else reversegen(tail(l) 

cons(head(l) m )). 

The generalization mechanism and the extended proce¬ 
dure-formation principle are just beginning to be formulated; 
the elaboration of these concepts is an important part of our 
projected effort. Generalization was proposed as a program- 
synthesis technique by Siklossy, 4 and is routinely performed 
by theorem-proving systems for proofs by induction (e.g., 
see Boyer and Moore 5 ). 

STRUCTURE-CHANGING PROGRAMS 

In the discussion so far, we have been concerned with 
structure-maintaining (i.e., “side-effect-free”) programs, 
which produce an output but effect no permanent change in 
the data objects of the programming environment. To pro¬ 
duce structure-changing programs, which can effect such 
changes, we introduce structure-changing primitives, such 
as assignment statements, into our target language. To spec¬ 
ify such programs, we admit new constructs into our spec¬ 
ification language. Most notably, we employ achieve P to 
denote a program intended to make the condition P true. 

All the principles we have applied to construct structure- 
maintaining programs may be adapted to construct struc¬ 
ture-changing programs as well. However, certain new prob¬ 
lems arise in the synthesis of structure-changing programs; 
among these is the simultaneous-goal problem. 

In constructing a program to achieve two conditions Pi 
and P 2 , it is not sufficient to decompose the problem by 
constructing two independent programs to achieve Pi and 
P 2 respectively. The program that achieves P 2 may in the 
process make P t false, and vice versa. Thus, the concaten¬ 
ation of the two programs will not achieve both conditions. 

For example, suppose we want to construct a program to 
sort the values of three variables x, y, and z; in other words, 
we want to permute the values of the variables to achieve 
the two conditions and y=sz simultaneously. Assume 
that we are given the primitive instruction sort2(u v ), which 
sorts the values of its input variables u and v. Then we can 
achieve each of our desired conditions independently by 
executing the program segment sort2(x y) and sort2(y z) re¬ 
spectively. However, the concatenation 

sort 2(x y) 

sort2(y z) 

of these two segments will not achieve both conditions si¬ 
multaneously; in sorting y and z, the second segment 
sort2(y z) may make the first condition x<y false. 


To circumvent difficulties of this sort, we have introduced 
the following simultaneous-goal principle: To satisfy a goal 
of form 

achieve Pi and P 2 , 

first construct a program F to achieve Pi, then modify F to 
achieve P 2 while protecting Pi at the end of F. The program- 
modification technique we employ is based on the “weakest- 
precondition operator” (Dijkstra 6 ). A special “protection 
mechanism” (cf. Sussman 7 ) ensures that no modification is 
permitted that destroys the truth of the protected condition 
Pi at the end of the program. 

To apply this principle to the goal 

achieve x^yand y<z 

in the sorting problem, we first construct the program seg¬ 
ment sort2(x y) that achieves the first condition. We then 
modify this program to achieve the second condition y^z. 
We cannot achieve this condition by inserting the instruction 
sort2(y z) at the end of the program, because (as we have 
seen) this modification violates the condition x<y, which we 
must protect. 

However, our program-modification technique allows us 
to achieve a goal by inserting modifications at any point in 
the program, not merely at the end. In this case, the tech¬ 
nique causes us to introduce the two instructions 

if y<x then sort2(x z) 

and 

ifx^y then sort2(y z ) 

at the beginning of the program segment. The modified pro¬ 
gram, 

tf y<x then sort2(x z) 
ifx^y then sort2(y z) 
sort2(x y), 

will achieve both conditions and ysz simultaneously. 

Similar techniques are employed by the system of War¬ 
ren. 17 

The derivation of straight-line programs with simple side- 
effects is now fairly well understood; much work needs to 
be done on the derivation of structure-changing programs 
with conditional expressions and loops, and the derivation 
of programs that alter list structures and other complex data 
objects. 

APPLICATIONS TO PROGRAMMING 
METHODOLOGY 

Although the development of a practical program-synthe¬ 
sis system requires considerable research effort, certain ap¬ 
plications of program-synthesis techniques to more re¬ 
stricted problems will be of more immediate practical value. 

Various programming disciplines have been evolving that 
attempt to make the programming process simpler or more 
reliable. Let us consider several of these disciplines, to see 
where program-synthesis techniques may be applicable. 
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• Structured Programming —Like program synthesis, 
structured programming (cf. Dijkstra 8 ) presents princi¬ 
ples for deriving a program systematically from given 
specifications. However, the principles of structured 
programming are intended to guide a human program¬ 
mer, whereas the principles of program synthesis are 
meant to direct a computer system. Nevertheless, we 
have found that some of the techniques we have de¬ 
veloped for a program-synthesis system could well be 
employed by a human programmer. In particular, the 
recursion-formation principle is a better motivated 
guide for introducing a loop than the conventional struc¬ 
tured-programming method for the same task. 

• Program Transformation —In this approach (cf. Bur- 
stall and Darlington 3 ), the programmer constructs a 
transparent program for his task, which is likely to be 
correct but which may be inefficient. This program is 
then transformed into an efficient equivalent program, 
which may be more difficult to understand. This trans¬ 
formation process is guaranteed to produce a program 
equivalent to the original. Program transformation may 
be regarded as a synthesis task in which the specifica¬ 
tions are given in the form of a clear program in the 
target language. All the program synthesis techniques 
we have developed can be applied to program transfor¬ 
mation as well. 

• Data Abstraction —In this approach, the programmer 
expresses his program in terms of abstract data types, 
objects such as sets, queues, or graphs whose proper¬ 
ties are well-defined but whose precise machine repre¬ 
sentation is left unspecified. When this program is com¬ 
plete, representations for its abstract data types are 
chosen and the program is transformed to replace the 
operations on the abstract data types by the corre¬ 
sponding concrete operations on the chosen represen¬ 
tation (cf. Guttag et al. 9 ). 

• Program Modification —It is often observed that pro¬ 
grammers spend more of their time extending programs 
that already perform some task correctly than they do 
in developing new programs. This process is particu¬ 
larly fraught with error, because in modifying a pro¬ 
gram, the programmer is likely to make some change 
that interferes with the program’s original functioning. 
We have remarked that a program-modification tech¬ 
nique was developed to support the simultaneous-goal 
principle. This technique can also be applied to perform 
independent program-modification tasks. The protec¬ 
tion mechanism ensures that the modified program must 
still perform the task for which it was originally in¬ 
tended. 

IMPLEMENTATION 

It is difficult to develop or evaluate heuristic techniques 
without experimenting with an implementation. The DE- 
DALUS system is a laboratory tool rather than a practical 
product. The system is implemented in QLISP (Wilber 10 ), 
an extension of INTERLISP (Teitelman 11 ) that includes pat¬ 


tern-matching and backtracking facilities. In this section, we 
will describe some of the special characteristics of our im¬ 
plementation without going into very much detail. 

The specifications are expressed in a LISP-like notation. 
Thus, the output specification for the lessall program, which 
we wrote as 

x<all(l), 

is represented in the DEDALUS system as 
(LESS X (ALL L)). 

The output specification for the gcd program, which we 
wrote as 

max{z: z\x and z|y}, 
is represented as 

(MAX (SETOF Z (AND (DIVIDES Z X) 

(DIVIDES Z Y)))). 

The target program is also expressed in LISP syntax. 

The transformation rules are expressed as programs in the 
QLISP programming language. For example, the rule that 
we denoted by 

Q and true Q 

is represented by the QLISP program 
(QLAMBDA (AND <^Q TRUE) $Q). 

The rule 

u\v true if u is an integer and v=0 

is expressed in QLISP as 

(QLAMBDA (DIVIDES ^V) 

(INSIST (PROVE ('(INTEGER $U)))) 
(INSIST (PROVE ('(EQUAL $V 0)))) 
TRUE). 

Although the reader who is unfamiliar with the QLISP lan¬ 
guage may not understand all the details of the above pro¬ 
grams, he may still observe that they are similar in form to 
the rules that they represent; the features of the QLISP 
language make this representation fairly direct. Because 
rules are represented as programs, we are allowed the full 
power of the programming language in expressing each rule. 

The DEDALUS system currently contains more than a 
hundred such transformation rules. In expanding the system 
to handle a new subject domain, we simply introduce new 
rules. 

The rules of the system are classified according to their 
pattern, their left-hand side. This pattern describes the class 
of subgoals to which the rule can be applied. Thus, the rules 

u\v =£> true if . . . 

and 

u\v =£> u\v—u if . . . 

both have pattern u\v, and can be applied to goals such as 
compute x\y+z. 
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When a new goal is generated, the QLISP system retrieves 
those rules whose patterns match the form of the goal. This 
retrieval is facilitated by arranging the rules in a classifica¬ 
tion tree according to their patterns; thus the two rules 
above would be classified on the same branch of the tree. 
This mechanism allows us to avoid matching every rule in 
the system against each newly-generated goal. 

If no rule matches the entire expression of a goal, its 
subexpressions are established as subgoals. If no rule 
matches any subexpression of a given goal, a failure occurs, 
and backtracking is invoked; the system attempts to find an 
alternate transformation that applies to a previous subgoal. 

The QLISP pattern-matcher has special provisions for 
matching commutative functions. Thus, because the and 
operation is commutative, the rule 

Q and true => Q, 

represented as the QLISP program 

(QLAMBDA (AND ^Q TRUE) $Q), 

can be applied to goals of form “true and Q” as well as “Q 
and true". For this reason, commutativity rules such as 

P and Q => Q and P 

are not necessary in the DEDALUS system. 

This kind of matching also occurs in the recursion-for¬ 
mation rule, in determining whether a new goal is an instance 
of some earlier goal. For example, in the actual synthesis of 
the gcd program, the top-level goal 

compute max{z: z\xand z\y} 

is regarded as an instance of itself with the roles of x and y 
reversed, because the and function is commutative. The 
recursion-formation rule, therefore, is able to propose the 
recursive call gcd(y x ). 

The final gcd program we obtain is 

gcd(x y)^ify<x 

then gcd(y x) 
else if x 4-0 

then gcd(x y—x) 
else y. 

Currently, the DEDALUS implementation incorporates 
the principles of conditional formation, recursion formation 
(including the termination proofs), and procedure formation, 
but not generalization or the formation of structure-changing 
programs. The techniques for deriving straight-line struc¬ 
ture-changing programs were implemented in a separate sys¬ 
tem (see Waldinger 12 ). 

Representative samples of the programs constructed by 
the current DEDALUS system are the following. 

Numerical Programs; 

• the subtractive gcd algorithm 

• the Euclidean gcd algorithm 

• the binary gcd algorithm 

• the remainder of dividing two integers 


List Programs: 

• finding the maximum element of a list 

• testing if a list is sorted 

• testing if a number is less than every element of a list 
of numbers ( lessall )) 

• testing if every element of one list of numbers is less 
than every element of another 

Set Programs: 

• computing the union or intersection of two sets 

• testing if an element belongs to a set 

• testing if one set is a subset of another 

• computing the Cartesian product of two sets (cart). 


COMPARISONS WITH AUTOMATIC PROGRAMMING 

It has been claimed (e.g., see Balzer 13 ) that, for a complex 
programming task, it is unrealistic to expect the user to 
formulate complete, correct specifications for the desired 
program. In specifying an airline-reservation system, an op¬ 
erating system, or a spacecraft-guidance system, for exam¬ 
ple, we are unlikely to anticipate the desired behavior of the 
system in every possible situation. In some systems, the 
specifications for the program are formulated gradually 
through an extended dialogue between the user and the 
system. (See, e.g., Green 14 and Balzer et al., 15 or the survey 
of Heidom 16 .) The dialogue is continued during the program- 
construction process, so that the user can aid in the design 
of the algorithm and resolve any ambiguities or inconsis¬ 
tencies the system might discover. Typically, these systems 
attempt to play the role of an expert programmer-consultant, 
and they tend to rely more on built-in or acquired knowledge 
of a particular subject domain than on deductive processes. 
By concentrating on basic programming principles, we have 
focused on general techniques that apply to any subject 
domain. The ultimate automatic programming system, of 
course, will combine specific subject knowledge with gen¬ 
eral deductive methods. 
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Automatic representation selection for 
associative data structures 
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INTRODUCTION 

Motivation 

One of the persistent hopes for rapid progress in Computer 
Science is that standard solutions to software design prob¬ 
lems will be developed, analyzed, and used. In particular, 
the hope for high-level languages has always been that 
enough low-level detail could be left to “the system” to 
allow a mere human to develop solutions for enormously 
complex problems. An ideal high-level language would allow 
a programmer to concentrate on the conceptual development 
of a solution for his problem, rather than on the design of 
a representation for his solution once he can express it. 

Although existing high-level languages insulate a user 
from many details, they still require him to do low-level 
design for many problems, especially for complex ones. 
Such cases arise either because the language provides only 
low-level storage structures (records, arrays, pointers, lists) 
with which the programmer must encode his algorithm, or 
because it provides a general-purpose data structure for 
which an arbitrary (and usually sub-optimal) representation 
design choice has been made a priori (LISP, 1 SAIL 2 ). Any 
single representation choice for a general data structure is 
very badly matched to some problems; the user of a general 
data structure must often spend much time designing a way 
to avoid unacceptable inefficiencies. Furthermore, good 
storage structure design requires much clerical work, for 
which a person whose goal is to solve a problem will usually 
have little time or enthusiasm. An automatic system, on the 
other hand, is well suited to the job of systematically com¬ 
paring the cost and applicability of the many representations 
that are available for program data. In addition, an automatic 
system is less likely than a person to overlook possible 
representation candidates. 

The problem 

This paper explores one aspect of the representation se¬ 
lection problem: storage structure selection for associative 
data structures. By “associative data structure,” we mean 
a collection of associations between abstract items. We ex¬ 


press associations as ordered lists (“tuples”) of items. Thus, 
an association between n given items is (conceptually) an n 
element list of them, called an “n-tuple”. The role of each 
item in an association is identified by its position in the n- 
tuple. 

Before going into specific detail about representation se¬ 
lection, we present a simple example which will be used to 
illustrate technical details as they arise. The example data 
base is a collection of relations among family members, their 
genders, and their places of residence. For the most part, 
we will be concerned with three-part associations (triples) 
such as: 

Sexof • Abby=Female 

Parentof • Eric=Roberta 

or abstractly A-0=V. The A,0,V notation comes from con¬ 
sidering the association as: 

Attributeof-Object=V alue 

We view a collection of associations in the computer as 
a software “associative memory.” 3 That is, one could re¬ 
trieve any (3-part) association by specifying any one or two 
of its positions. Figure 1 describes these possibilities. 

An associative language will contain operations like: 

Make Parentof-Eric=Paul 
and 

Erase Homeof-ANY=Valhalla 

which add or delete associations or sets of associations from 
the collection. There will also be operations to retrieve and 
make use of associative information like: 

Print (Sexof-Peggy) 
and 

Foreach PERSON such that 
Homeof-PERSON=Rochester do 
Print (PERSON, “lives in Rochester”). 
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Form 

Example 

Interpretation 

A-0=V 

Homeof - Paul=V alhalla 

The association itself, if present 

A-0=X 

Parentof-Eric=PARENT 

Parents of Eric 

A-X=V 

Parentof-CHILD=Paul 

Children of Paul 

XO=V 

RELATION -Eric=Rochester 

Attributes linking Eric to Rochester 

A-X=Z 

Parentof-CHILD=PARENT 

Child-Parent pairs present 

X-Z=V 

RELATION -PERSON=Male 

All pairs of attributes and objects yielding 

Male 

XO=Z 

RELATION -Abby=VALUE 

All attributes of Abby and their values 


Figure 1 


In the latter case, PERSON is a variable that is repeatedly 
“bound” to the middle (Object) position of associations 
matching the pattern. 

We are interested in how to implement a programming 
system that can do such operations efficiently. The diffi¬ 
culty, of course, is that each program is different and one 
cannot expect a single implementation to be best for all 
programs. What is required is an “intelligent compiler” 
which selects a different collection of storage structures for 
each program. The first such effort 4 was concerned primarily 
with sets and lists, which are simpler to deal with than 
associations. This paper explores the automatic selection 
problem for associations, and describes an experimental sys¬ 
tem (called the “selection system”) which embodies our 
ideas. The discussion centers around associative data struc¬ 
tures which fit into some “main” memory. (The problem 
for structures in several different kinds of memory is more 
complex.) 

The following discussion illustrates the behavior of the 
system using the following simple program as an example: 


BEGIN 

Attributes Parentof, Sexof, Homeof 
Item Constant Male, Female, Peggy, Abby, Rochester,... 
Item Variable CHILD, PARENT 
Make Parentof-Abby=Paul 
Make Parentof-Abby=Roberta 
Make Sexof-Abby=Female 
Make Homeof • Abby=Rochester 
Make Homeof-Thor=V alhalla 


(1) Foreach CHILD, PARENT such that 
Sexof-CHILD=Male 
and Parentof-CHILD=PARENT do 
Print (“Sonof” PARENT “is” CHILD) 

(2) Foreach CHILD, PARENT such that 
Parentof-CHILD=PARENT 
and Homeof-PARENT=Rochester do 
Print (CHILD “writes home to Rochester for 
money”) 

END 

Figure 2 


First let us consider the even simpler program with the 
part in the box omitted. The program builds a collection of 


associations involving the three attributes. The “Foreach” 
part computes (parent,son) relationships in two steps. First 
it chooses an association of the form 

Sexof-0=Male 

and binds CHILD to the value of “O” in the association. It 
then uses that value to match an association of the form 

Parentof CHILD=V 

and assigns the matching V to Parent, and prints. This is 
repeated until there are no more matching associations. We 
will see how the system chooses storage structures for this 
simple program. 

A MODEL OF ASSOCIATIONS AND HOW THEY ARE 

USED 

How should a program (in an associative language) be 
described for the purpose of designing a storage structure 
for its associative data structures? The first thing to note is 
that a program usually deals with more than one class of 
association. A class of associations is a group which will be 
represented the same way internally. In the example, there 
are two classes: one based on the Parentof relation and one 
on the Sexof relation. Since there are no searches on the 
Homeof relation (with the boxed statement omitted), the 
selection system would not include those associations at all. 
For each class of associations, the system will characterize 
the operations performed on that class and the possible 
values for unbound variables. In our simple case, each class 
has several MAKE operations and one SEARCH. 

The differences among classes of associations are often 
reflected in the ways that the program accesses and modifies 
them. That is, a program often uses different parameter 
patterns for associative operations on different classes of 
associations. For example, the simple program includes a 
search for male children, i.e., 

(given: Sexof)-(variable to be bound) = (given: Male) 

but not for the sex of a given child, i.e., 

(given: Sexof)-(given: Eric)=(variable to be bound) 
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For the “Parentof” association, on the other hand, there 
are two SEARCH parameter patterns (with the statement in 
the box included): 

(given: Parentof)-(given: a child)=(variable) 
and 

(given) • (variable) = (variable) 

In another case, a program may require access to attri¬ 
butes that relate two items, for example spatial relations 
between two objects in a 3-D scene: above, below, behind, 
or in-front-of. 

Other reflections of the differences in associations are to 
be found in properties of the associations themselves. For 
example, the sex of a person has only one value, but a 
parent may have several children. 

Often, the conceptual differences in the associations with 
which a program deals provide information that is useful for 
making low-level representation decisions. For example, a 
representation that immediately comes to mind for the Sexof 
attribute is a pre-defined (at compile-time) field of every 
PERSON record, to hold a code (one bit is enough) which 
specifies Male or Female. 

If the program used only searches for the sex of a given 
child, such a representation would be just fine. But the 
program does not use the Sexof relation this way, but instead 
asks that people who are Male be enumerated. The above 
representation would require that all PERSON records (both 
male and female) be scanned for ones that are marked 
“Male”. For this purpose, a list of male people might be a 
better representation. This example points out that classes 
of associations should be characterized both in terms of the 
operations of the program and the properties of the data. 

Now that we have an intuitive understanding of what 
“class of associations” means, we will define it more pre¬ 
cisely, and sketch how the automatic selection system de¬ 
termines the different classes of associations that a given 
program uses. 

The process begins with a flow analysis phase, in which 
the system determines the possible run-time values of item 
variables at their points of use in the program, and the 
possible associations that could exist at run-time. The first 
approximation to the collection of classes of associations is 
one class for each MAKE statement, representing those 
associations that the MAKE statement would create (at run¬ 
time). For our simple example, each such class would con¬ 
tain one triple. 

The next phase causes two classes to merge if any single 
SEARCH or ERASE operation could access associations 
from both classes. The basic idea is that each SEARCH or 
ERASE operation divides the store of triples into two sub¬ 
sets: those triples which could match the parameter pattern 
of the operation (subset A) and those triples which could 
not (subset B). If the A subsets for two operations intersect, 
the algorithm merges the subsets, and associates both op¬ 
erations with the result. This continues until there are no 


more mergers to perform. The result is a model of the (dis¬ 
joint) sets of triples (called “classes of triples”) that will 
exist when the program is run. Each class has an associated 
set of operations which access only the triples of the class. 
The use of this algorithm reflects an implicit assumption 
about computation cost: a representation that would require 
access to more than one storage structure to service a single 
associative retrieval operation is assumed a priori to be too 
expensive. For our simple example (with the statement in 
the box included), the algorithm would yield three classes 
of associations: one for each of the three attributes. 

A LIBRARY OF REPRESENTATIONS 

The automatic selection system knows how to deal with 
roughly twenty different storage structure representations, 
but these are all based on a few fundamental concepts. One 
representation of a set of triples is simply to store triples in 
a vector of entries three items wide and to search all entries 
for the answers to the questions in Figure 1. There are a 
variety of record techniques in which a fixed location in a 
block of storage is associated with an attribute, e.g., one 
could have records for people with a Sexof field, etc. An¬ 
other common technique is to form a linked list of attribute- 
value pairs for a given object (a “property-list”). Other 
useful ideas include inverted lists (a link through all tuples 
which contain a given item), hash-coding, 5 and using one- 
bit fields to represent the presence or absence of a property. 

The selection system has a built-in library of representa¬ 
tions for associations. The library is organized in two parts: 
“associative retrieval techniques” for servicing individual 
associative operations, and “associative representations” 
for common combinations of operations. The following dis¬ 
cussion introduces the retrieval techniques and storage 
structures that provide access to triples, and describes the 
ways in which the storage structures can be varied and 
combined to synthesize “representations.” This two-level 
organization is useful for proposing candidate representa¬ 
tions: The analysis of which retrieval techniques are appli¬ 
cable to each associative operation on a class of triples need 
not be re-done for each (composite) representation that is 
considered. For example, once hash-coding is eliminated as 
a possible retrieval technique for an operation, no represen¬ 
tations which would use hash-coding for the operation will 
be tried. Such methods for avoiding duplication of analysis 
effort are crucial for controlling combinatorial explosion in 
the selection process. 

We have chosen four basic retrieval techniques upon 
which to base the example representations in the library: 
field selection records, property lists, inverted files, and 
hash tables. The diagrams in Figure 3 illustrate simple stor¬ 
age structures which implement these retrieval techniques. 

The library consists of twenty-three prototype represen¬ 
tations which are based on these simple storage structures. 
We sketch these below: for a complete discussion, see Ref¬ 
erence 6. 

For the purpose of the discussion below, we will assume 
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1. FIELD SELECTION RECORD 



2. PROPERTY LIST SEARCH 


3. 



INVERTED FILE SEARCH 


ATTRIBUTE: 

(A x ) 


4. HASH TABLE LOOK-UP 
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that each item has a unique (integer) “internal name.” We 
will discuss using this integer as a hash-code operand, or as 
a key to various useful information about the item. Specif¬ 
ically, we will assume that the internal name of an item is 
a pointer to a block of contiguous storage cells, and refer to 
such blocks as “item descriptors.’’ A typical item descriptor 
contains space for bit vectors and pointers to various lists 
that the selection system might allocate for triples that con¬ 
tain the item. 

Field selection records 

There are three variations, corresponding to the cases 
where the value field of the triple is binary (one bit), single¬ 
valued (e.g., the Homeof attribute), and multiple-valued. 

Complex inverted list 

This representation provides one sequential block of stor¬ 
age cells in memory for each triple (called “triple block”), 
with one field for each item. Each block can be threaded on 
either one, two, or three lists: one list for each item in the 
triple. The head of each such list is contained in an item 
descriptor. 

P-List 

A P-list is a list of (attribute-value) pairs. The head of 
each list is contained in the item descriptor for each object. 
There may be more than one value for each (attribute,object) 
pair. 

Two argument hash-table 

Two items (internal values of the attribute and object) are 
hash-coded (using a bucket hash) to locate a cell in a hash- 
table; the cell contains the attribute, the object, and a list of 
values. The cell can be threaded in either one or two lists: 
one list for each hash operand. The header for each such 
list resides in the item descriptor for the item which is the 
hash-operand. This variation is effectively a combination of 
a hash-coding retrieval technique and either one or two P- 
list retrieval techniques. 

Another variation of the hash-table representation is to 
thread each value in an inverted list (of triples which have 
that value). Each of these representations combines a hash¬ 
coding retrieval technique with an inverted file retrieval 
technique. 

A third variation is to use entries in the hash table to point 
to triple blocks, which can be threaded in a complex inverted 
list. 

Three argument hash-table 

The three item internal names are used as hash-coding 
operands. Each cell in the hash table represents one triple 


either by storing the three items, or by pointing to a triple 
block (which can be threaded as above). In addition, each 
cell can be threaded in one, two, or three inverted lists. 

The above representations allow great flexibility in com¬ 
bining property-list, inverted list, and hash-coding retrieval 
techniques. This flexibility is most useful for dealing with 
ERASE operations, which must delete all paths to a triple 
in a complex representation when the triple is erased. In 
particular, certain of the above representations support 
back-pointers in their internal lists. The selection system 
makes use of this information. A more thorough discussion 
of the details is beyond the scope of this paper; the reader 
is referred to Reference 6. 

Each retrieval technique in the library is associated with 
an “applicability” rule. This is an heuristic which is used to 
determine whether the access structure should be used to 
realize a given ERASE or SEARCH operation. Applicability 
rules usually reflect “common sense” (to a storage structure 
designer) rules, and are employed to limit combinatorial 
explosion in the selection process. These are useful con¬ 
straints that represent pre-canned cost analysis decisions, 
like “blind search should be avoided whenever possible”. 
Applicability rules are also used to detect relatively common 
special cases for which an unusual representation is partic¬ 
ularly well suited. 

For example, let us consider the basic record-style rep¬ 
resentation of a triple. A typical choice for the sample uni¬ 
verse described above would be to have a record for each 
person with fields for his Sex, Home, and Parents. Since 
there may be two parents, the Parentof field might point to 
a set of parents, yielding something like the following struc¬ 
ture for Eric: 


Homeof 

Rochester 


Sexof 

Male 


Parentof - 

-> 



There are additional complications, such as how to represent 
the set of parents. More to the point, this representation will 
not support an associative search such as: 

Parentof CHILD=Paul 

without exhaustive search. The applicability rule for the 
record-style access structure would eliminate it as a candi¬ 
date for the Sexof association because of the need for ex¬ 
haustive search. Figure 4 contains a partial list of the com¬ 
putation cost tradeoff assumptions that underlie the heuristic 
applicability rules. 

It is the nature of heuristic search that it is always possible 
to concoct situations in which a path that would turn out to 
yield an optimal solution is left untried. As for any heuristic 
method, the use of applicability rules is meant to achieve a 
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1. It is cost effective to maintain ordered lists of VALUE items in instances 
of the following access structures: 

Field Selection Records 

Property lists 

Two Operand Hash Tables 

2. It is not worthwhile to consider an associative retrieval algorithm which 
requires that all items that could appear in a given operand position for 
a given class of triples be enumerated and tested. 

3. The bookkeeping overhead for dealing with storage blocks for Field 
Selection Records whose lengths might vary at run-time is not cost ef¬ 
fective. 

4. It is cost effective to maintain an ordering relation on the elements of a 
property list access structure. 

5. It is not worthwhile to consider an associative retrieval algorithm which 
requires the examination of all triples in a property list access structure 
for a given item. 

6. It is cost effective to maintain an ordering relation on the elements of an 
inverted list access structure. 

7. It is not worthwhile to consider an associative retrieval algorithm which 
requires the examination of all triples in an inverted list access structure 
for a given item. 

8. It is cost effective to use the necessary storage space and execution time 
to maintain explicit “back pointers” in an inverted list access structure 
if it would otherwise be necessary to search the list from its beginning to 
find the list element which precedes a given one. 

9. A hash coding scheme which uses chaining for collisions is as good as 
any other hash coding scheme. 

10. It is not worthwhile to consider an associative retrieval algorithm which 
requires preliminary searching to locate the buckets in a hash table which 
might contain answers. 

Figure 4 


compromise between optimal solutions at tremendous ex¬ 
pense and poor solutions that are easy to find. 

STRATEGIES FOR PROPOSING REPRESENTATIONS 
FOR THE TRIPLES OF A GIVEN PROGRAM 

Proposing non-redundant representations 

This section describes how the selection system uses the 
model of a class of triples and the information in the library 
of representations to propose storage structures to represent 
the class. Proposal proceeds in two phases. First, each op¬ 
eration is analyzed to determine a set of applicable tech¬ 
niques. Next, an attempt is made to find representations 
which provide an applicable technique for each operation of 
the class. Heuristic rules are used to eliminate inefficient 
representations. 

For each operation on the class, each technique in the 
library is examined. If its applicability criterion is satisfied 
for the operation, it is added to the set of techniques that 
are applicable to that operation. A technique may apply to 
an operation in more than one way. For example, suppose 
we had a SEARCH operation of the form: 

(given) • (given)=(variable) 

(like Parentof-(given child) = (variable to bind)) 

and we were considering the inverted file technique. We 


could either keep a list of triples for each item which appears 
as ATTRIBUTE (e.g., Parentof), or we could keep a list of 
triples for each item which appears as OBJECT (e.g., child). 
In the first case, we would use the given ATTRIBUTE to 
locate the list, and then search it for triples which have the 
given OBJECT. In the second case, we would use the OB¬ 
JECT to locate the list, and then search it for triples which 
have the given ATTRIBUTE. Locating such an inverted list 
of triples is a simple computation: the pointer to the list 
would be contained in a known field of the ATTRIBUTE’S 
(OBJECT’S) item descriptor. The internal name of an item 
is a pointer to its item descriptor; the selection system would 
assign one field of the descriptor to contain the pointer to 
the inverted list. 

In the above example, the two options are represented by 
the selection system as a set of permutations of the triple 
operand pattern, as follows: 

{(AOV),(OAV)} 

The convention for denoting variations of the inverted file 
technique is that the first letter in a permutation indicates 
the position by which the list is accessed, and the second 
indicates the position by which the list is ordered. The two 
options here are “the list is accessed by A (i.e., ATTRI¬ 
BUTE), and ordered by O (i.e., OBJECT),” and “the list is 
accessed by O, and ordered by A.” For our simple example, 
these correspond to keeping a single list of all (child,parent) 
pairs, ordered by the internal name of the child, and keeping 
an unordered list for each child of his parents. 

After each operation in a class is associated with all tech¬ 
niques that can be used to realize it, the second phase of 
proposal occurs: finding candidate storage structures that 
provide techniques for all operations of a class. In our ex¬ 
ample (with the statement in the box included), the Parentof 
class has two search operations, with the following forms: 

(given)-(given)=(variable) 
and 

(given )• (variable )=(variable) 

A slightly more complex case is a class with several op¬ 
erations where all operations have the same set of applicable 
techniques, and for each of these techniques the same set 
of permutations. This will occur, for example, if all opera¬ 
tions of the class have the same form (i.e., the same pattern 
of (given),(variable), or ANY parameter descriptions). In 
such a case we treat the set of operations as a single oper¬ 
ation, and the task is the same as described above. All 
operations of the class will use the same associative retrieval 
technique. 

In the general case, there are several forms of SEARCH 
operations which retrieve associations, and the sets of ap¬ 
plicable techniques are not the same for all operations. Even 
in such a case, the intersection of these sets may not be 
empty. That is, there may be at least one technique which 
is applicable to more than one operation. The general prob- 
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lem is to find ways to decompose a set of operations into 
subsets each of which can be serviced by one associative 
retrieval technique. 

Each representation in the library is cross referenced by 
the retrieval techniques that it provides. The algorithm for 
proposing representations for a decomposition is sketched 
below: 

Each representation which provides the required number 
of techniques is considered. Each applicable technique of 
each subset of operations in the decomposition is exam¬ 
ined. If it is a technique provided by one of the represen¬ 
tations, then the applicability rule for each such represen¬ 
tation is evaluated. The evaluation yields a (perhaps empty) 
set of “candidate representations” to be proposed. 

Each such “candidate representation” is a data structure 
which instantiates one of the representations in the library. 
That is, it specifies which retrieval techniques are to service 
which (subsets of) operations, and how (i.e., the operand 
permutation). In other words, an applicability rule for a 
representation specifies acceptable matches between the 
techniques that are provided by the representation and the 
techniques that are applicable to the given operations. 

To avoid duplication of effort, the selection system treats 
similar candidate representations as a unit whenever possi¬ 
ble. Specifically, candidate representations which differ only 
in their operand permutations are usually treated as a unit. 
For instance, the inverted file representation for the example 
above has two permutations which would be treated as a 
unit until final cost analysis is done. 

The system cannot always treat representations which 
differ only in their operand permutations as a unit. Aside 
from a few such special cases, however, differences in op¬ 
erand permutations are ignored until cost analysis is done, 
thus saving both space and time in the early phases of 
automatic selection. 


Proposing redundant representations 

The discussion above describes a method for proposing 
candidate representations for a class of triples where each 
triple in the class is stored in memory only once. In such a 
representation, each triple will be represented by either one 
triple record (with perhaps several threads), or one VALUE 
cell (perhaps in a list of them), or one field of an item record. 

The selection system also considers representations in 
which triples are stored more than once. The advantage of 
keeping multiple copies is that each copy can serve a special 
purpose for which its storage structure can be custom de¬ 
signed. In a case where there is no single representation 
which will service all the operations on the class efficiently, 
there may be a way to decompose the set of operations into 
subsets for which efficient special purpose representations 
can be found. It is more likely that an efficient storage 
structure can be found for a small set of associative opera¬ 
tions than for a large set. Depending on the cost tradeoff 
between storage space and execution time, such a decom¬ 


position might be “cheaper” than a monolithic representa¬ 
tion, even though more storage space is required. For ex¬ 
ample, a membership relation (“ Memberof-Set=Element”) 
between sets and their elements might be represented both 
as a list for each set of its elements and as a list for each 
element of the sets to which it belongs. Such a representa¬ 
tion would provide speedy access both for queries about the 
elements of a given set: 

Memberof-(given)=(variable to bind) 

and for queries about the sets to which a given element 
belongs: 

Memberof-(variable to bind)=(given) 

To propose redundant representations, the selection sys¬ 
tem uses the method for proposing non-redundant represen¬ 
tations as a subroutine. The problem is to discover decom¬ 
positions for which efficient redundant representations can 
be found. The method is based on enumerating the ways of 
partitioning the set of ERASE and SEARCH operations on 
a class of triples. Roughly, we look for ways to decompose 
the set of operations on a class into a collection of sub¬ 
classes, each of which can be analyzed for applicable storage 
structure representations (using the methods described 
above) as if it contained the only operations on the class. 
The idea is to capture in the notion of “subclass” a way to 
identify groups of operations for which efficient storage 
structures exist. The result of the analysis of a given class 
of triples is a collection of candidate redundant representa¬ 
tions, each having one “efficient” (see below) storage struc¬ 
ture for each subclass in one partition. Thus, for a particular 
partition (i.e., set of n subclasses), each proposed represen¬ 
tation requires n copies of each triple: one copy in each of 
n storage structures. Each MAKE operation for the class of 
triples must create its triple in each of the n storage struc¬ 
tures, and each ERASE operation must remove triples 
which match its operand pattern from all n storage structures 
(problems of critical races and deadlock do not arise). 

In our example of a redundant representation for a set 
membership relation, each association would be stored 
twice: once as an entry on the list of elements in the given 
set, and once as an entry on the list of sets which contain 
the given element. 

In any representation, each execution of a SEARCH op¬ 
eration must find all triples which satisfy the operand pattern 
of the SEARCH. We assume that each SEARCH operation 
is associated with exactly one subclass. That is, it will find 
all its answers from one storage structure. Thus, each sub¬ 
class must service all MAKE operations for the class. For 
consistency, each storage structure will contain all the tri¬ 
ples of the class (one could imagine clever ways to relax this 
requirement, but we do not consider them here). 

Unless the rules for acceptable redundant representa¬ 
tions are more restrictive than the rules for non-redundant 
representations, the number of candidate representations 
will be very large, and the cost of the overall analysis 
will be huge. There is always a way of using our library and 
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applicability rules to design a storage structure to service a 
given set of associative operations. If everything else fails, 
a complex inverted list representation could be used, with 
a separate thread for each operation. Thus, every possible 
subclass would be acceptable, and each partition of the class 
of triples would be plausible. If several representations were 
applicable to each subclass, things get worse. If we consider 
the fact that the number of applicable redundant represen¬ 
tations for a partition is the product of the numbers of 
representations for its subclasses, we can begin to see that 
unless the applicability rules are tightened for subclasses, 
there could be very many candidate redundant representa¬ 
tions indeed. 

To limit the combinatorial explosion of candidate repre¬ 
sentations that use multiple copies of the store of triples, we 
have chosen a specific notion of efficiency and an heuristic 
rule to decide when to propose such representations. In 
general, we consider a storage structure to be inefficient for 
an operation if significant searching is required to service 
the operation. Many of the cost tradeoff assumptions that 
are used by the selection system are based on this idea, and 
they provide (crude) thresholds for early detection of rep¬ 
resentations that are likely to be eliminated eventually in the 
detailed cost analysis. The most expensive kind of searching 
is “unrestricted” searching; it is this sort that the applica¬ 
bility rules disallow. For example, we do not even consider 
a record technique for a SEARCH operation which has only 
the ATTRIBUTE given; a search through all possible items 
which could be the OBJECT of a triple would be required 
to perform the SEARCH. It is convenient to distinguish four 
types of search: 

1. unrestricted 

2. loosely restricted 

3. tightly restricted 

4. fully restricted 

An example of “unrestricted” search is given above. By 
“loosely restricted” search, we mean (for example) search 
for elements which have a given attribute in a list of (attri¬ 
bute-value) pairs. By “tightly restricted” search, we mean 
(for example) search for a given item in a set of them (e.g., 
is Paul a Parentof Eric?). By “fully restricted” search, we 
mean access to answers via computation rather than search, 
e.g., selection of a given field of a given record, or hash- 
table look-up. 

The cost tradeoff assumptions used in proposing non-re- 
dundant representations lead to applicability rules which 
eliminate representations that require unrestricted search. 
Loosely restricted search is allowed for non-redundant can¬ 
didate representations, but not for redundant ones. In other 
words, 

We assume redundant representations which require 
loosely restricted search to be inefficient enough to be 
eliminated a priori as candidates for final selection. Intu¬ 
itively, it is only worth the price of multiple representa¬ 
tions if searching can be tightly (or fully) restricted. 


As redundant representations are synthesized for a class 
of triples, they are tested for loosely restricted search re¬ 
quirements. If such search is required, the representation is 
not proposed as a candidate. 

Now we consider the problem of proposing candidate 
redundant representations. The crude way to do it is to 
generate all possible ways of partitioning the set of SEARCH 
and ERASE operations of the class of triples. For each 
partition, each subclass would then be analyzed to determine 
applicable non-redundant representations. The cross prod¬ 
uct of the sets of representations for the subclasses of a 
partition would be the applicable redundant representations. 
The partition with one subclass (i.e., non-redundant) would 
fall out. 

The inherent symmetry of the problem leads to combi¬ 
natorial duplication of effort; early detection of situations 
that have already been analyzed is important. 

The analysis deals with subclasses, which are sets of 
SEARCH, ERASE, and MAKE operations, and partitions, 
which are (initially disjoint) sets of subclasses. Each asso¬ 
ciative operation corresponds to an expression in the source 
program, and is represented by the corresponding expres¬ 
sion node in the syntax tree of the program. 

The selection system uses a central data structure to keep 
track of the subclasses and partitions which it analyzes. This 
data structure (called the “set dictionary”) is similar in 
motivation to the “discrimination net” of QLISP: 7 it asso¬ 
ciates a unique descriptor with each set that is entered. 

The set dictionary is used by the selection system to avoid 
duplicating: 

1. The computation of applicable non-redundant repre¬ 
sentations for each unique subclass (i.e., set of oper¬ 
ations). 

2. The computation of redundant representations for each 
unique partition. 

Whenever a new subclass or partition is analyzed, it is 
entered into a set dictionary and the result of the analysis 
is recorded with the entry. A sketch of the method for 
proposing candidate redundant representations is presented 
below: 

A recursive function in the selection system generates 
the partitions of a set of operations, using another recur¬ 
sive function to generate the subsets (subclasses) of the 
given set. Each subset that is generated is looked up in 
the set dictionary. If not found, it is assigned a (new) 
descriptor, entered, and tested to determine whether ap¬ 
plicable representations exist (this test considers cost 
tradeoff assumptions for redundant representations. The 
new descriptor is associated with the result of the test. If 
it is found in the set dictionary, then it has already been 
tested. If the result of the test indicates no applicable 
representations exist, then the next subset is generated 
and the process continues. Thus, no subset is considered 
further unless it has applicable representations; i.e., once 
a subclass is analyzed and found to fail, no partition which 
contains the subclass will be considered. Furthermore, 
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because of the structure of the library, if a subclass fails, 
any subclass that contains it will fail. In other words, if 
there is no efficient representation for a given set of n 
operations, there will not be one for any larger set that 
contains the n operations. Thus, supersets of failed classes 
can be eliminated automatically. 

A method similar to the one described above for sub¬ 
classes is used for avoiding duplication of analysis effort for 
partitions. A partition is a set of subclasses. Each such set 
has a unique entry in the set dictionary. Thus, there are two 
kinds of entries in the set dictionary: sets of operations 
(subclasses) and sets of subclasses (partitions). After a par¬ 
tition is analyzed the first time, its descriptor is marked, and 
subsequent attempts to analyze it are short-circuited. 

The cost of avoiding duplication of analysis effort is the 
set dictionary look-up time; the large relative cost of ana¬ 
lyzing a subclass makes this overhead worthwhile. Another 
advantage of using the set dictionary is that storage space 
is conserved: only one cop^ of each unique set is kept, and 
only one copy of the analysis results is kept. 

HOW TO SELECT A GOOD REPRESENTATION 
FROM THE CANDIDATES 

In this section we show how to estimate the cost of each 
candidate representation and choose the cheapest one. We 
consider two components of computation cost: execution 
times (in micro-seconds) and storage space (in 36-bit words). 
For a given program, we are interested both in the execution 
time required to service its associative operations, and the 
storage space required for its associative data structures. 
We use three kinds of parameters to characterize each class 
of triples in a given program: 

1. the average sizes of substructures of the class of triples 
(e.g., For a given A and O, how many V’s will there 
be on average?). 

2. probabilities of occurrence of searches that fail and 
changes that are redundant. 

3. relative frequencies of statement executions. 

The computations for estimating the space and time costs 
of a given candidate representation for a given class of triples 
are based on evaluating formulas which are expressed in 
terms of these parameters. Each representation in the library 
is supplied with two functions: one for estimating storage 
space cost, and one for estimating execution time cost. The 
arguments to a space function specify a set of associative 
operations and for each operation the associative retrieval 
technique that will be used to realize it if the given repre¬ 
sentation is chosen. This information is used by the space 
function to identify the parameters for estimating storage 
space cost. 

A function for estimating time cost is a little more com¬ 
plicated. The arguments are the same, but the time function 
must sum the estimated contributions to total execution time 
of each of the operations. For each operation, this estimate 


is the product of two terms: the number of times the oper¬ 
ation was executed in the sample execution and the esti¬ 
mated cost of one execution of the operation for the candi¬ 
date representation. The estimated time cost of one 
execution of the operation is determined by a formula that 
is specific to the use of the indicated associative retrieval 
technique in the indicated representation. Each representa¬ 
tion in the library is provided with formulas for each of its 
associative retrieval techniques. There is one such formula 
for each associative operation to which a technique applies. 
Thus, for each operation, the arguments to the time function 
are used to select a formula and to identify necessary pa¬ 
rameters. 

For our example, the space cost formula for a record 
representation for Parentof and the time cost formula for a 
search operation of the form 

AO=X 

are shown in Figure 5. The space formula represents the 
requirement that there be a field of each record (0.5 cells) 
for each set-valued attribute (e.g., Parentof), and that space 
for the set of values is needed. In our example, there is only 
one attribute (Parentof), hence NA=1. NO (number of ob¬ 
jects) represents the number of people. NVALS(A,0) rep¬ 
resents the average number of parents per person. 

The flow chart for the search algorithm is also shown in 
Figure 5. The time cost formula represents the cost of the 
algorithm. Cl, C2, and C3 are constant terms; to a good 


Sample Cost Formulas for Records 
Space: NA*NO*(0.5 + SETSIZE (NVALS(A.O)) 

Time: SEARCH (A • O = X) 

Cl + C2 + C3 + PI * SGENCOST (NVALS(A.O)) 

Flow Chart for SEARCH (A • O = X) 

Cl: locate the record for O 

C2: locate the field for A within the 

record and pick up the pointer to the 
set of values . 

Y 

C3: is the set empty? j. 

f' x 

probability PI probability (1-PI) 

done 

SGENCOST(NVALS(A,0)): generate the elements 

of the set and return 
them one at a time 

done 


Figure 5 
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approximation they depend only on the machine and on 
basic design decisions and can be determined a priori. PI 
represents the probability that the search succeeds. The call 
on the SGENCOST function represents the execution time 
required to generate the elements of a set of a specified size 
(NVALS(A,0)). The values of PI, NO, and NVALS(A,0) 
are determined from answers to questions that the system 
asks. The choice of which questions to ask is based on 
which cost formulas are being used in a given run of the 
selection system. 

Parameters that depend only on the representations in the 
library (e.g.. Cl, C2, C3) are determined once when the 
library is defined, and built into the library. Their values are 
always available. Parameters that depend on the program 
are determined from information supplied by the program¬ 
mer in response to questions that the selection system asks. 

For example, the selection system asks three questions 
about the use of the Sexof relationship in our simple ex¬ 
ample. They are: 

1. How many males are there likely to be? Sample an¬ 
swer: 5. 

2. How often will the Sexof search fail to match? Sample 
answer: Never. 

3. How often will a redundant MAKE occur? Sample 
answer: Never. 

Not all parameters are needed: only ones required to eval¬ 
uate formulas that are relevant to the representation candi¬ 
dates for a class of triples. Moreover, some parameters can 
be computed from others. For example, if the answers to 

“For a given A, how many O’s (AVG) will have at least 

one V?” 

and 

“For a given A and O, what is the average number of 

V’s?” 

are known, then the selection system can deduce the average 
number of triples, since it can determine (by asking a ques¬ 
tion, if necessary) the average number of items that can 
appear in a triple of the class in a given position. Thus, any 
two of the above three answers determine the third. The 
system makes use of such dependencies to derive needed 
parameters from available ones if it can. If it can’t, it will 
ask a question. 

We should say a word here about our assumptions re¬ 
garding parameters. In any system that does automatic rep¬ 
resentation selection, a model of the expected behavior of 
the program is necessary. To make the analysis tractable, 
we made many intuitive decisions about the appropriate 
level of detail for cost formulas and their parameters. Two 
major assumptions about program behavior characterize our 
model and underlie our work on cost analysis: 

1. It is in general impossible to tell for an arbitrary pro¬ 
gram precisely how it will behave when it is executed. 


Instead, we depend on the relative frequencies of state¬ 
ment executions in one example execution, and on 
estimates given by the programmer about program be¬ 
havior and data structure size. We assume that the 
programmer has good enough intuitions about his pro¬ 
gram to provide a good general characterization. After 
all, he would have to use such intuitions if he were to 
design a representation without the help of an auto¬ 
matic system. 

2. To really do an optimal selection, we would need to 
use distributions of the values of the relevant param¬ 
eters both over time within a run and between'runs. 
We assume here that use of the average values of 
parameters (i.e., scalars) over the entire execution of 
the program provides enough information for good de¬ 
cisions about representation choice. 

In other words, we are using a crude estimate of computa¬ 
tion cost. This estimate should be useful for gross distinc¬ 
tions, but not fine ones. A human storage structure designer 
works much the same way: he often uses rough estimates 
of relative frequencies of operations and of relative sizes of 
data structure components to guide the choice of a storage 
structure. 

Because we are not making precise distinctions for esti¬ 
mating cost, it is not necessary to formulate precise char¬ 
acterizations of the associative search techniques and stor¬ 
age structures in the library. It is adequate to outline each 
algorithm and compose a cost formula for it in terms of the 
costs of its primitive steps, the parameters that describe 
average data structure size, and the parameters that describe 
expected program behavior. After enough experience with 
automatic selection systems like this one, we will learn where 
(heuristics, libraries, formulas, parameters, etc.) additional 
effort should go. 

After both space and time costs are estimated, if a rep¬ 
resentation is at least as expensive in both time and space 
as one already analyzed, it is eliminated as a candidate. 
Otherwise, its estimated overall computation cost is com¬ 
puted (see the discussion below), and it is added to an 
ordered list of candidates that have already been analyzed. 
The list is ordered by estimated overall computation cost. 
The final step in the current selection system is to print out 
detailed descriptions of the top few candidate representa¬ 
tions on the list. 

The choice for real programs of an appropriate overall 
measure of computation cost is a good research problem. 
Attempts at a solution might well lead to the development 
of new ways of describing programs and their behavior, and 
to new ideas and standards for the systems in which pro¬ 
grams function. Such a study is certainly worthwhile, but 
although we could make use of the results, it is outside the 
scope of this work and is left for future research. Instead 
(following Reference 4), we have provided in the selection 
system a function which defines the overall computation 
cost of a candidate representation to be the product of its 
storage space and execution time requirements. It would be 
easy to define a new function of space and time to replace 
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the one provided, or to experiment with several in the con¬ 
text of the selection system. 

Once cost estimates are derived for all candidates, we 
rely on the methods of Reference 4 to choose the best 
representation for the class of triples in the context of the 
entire program. In particular, we refer the reader to Low’s 
book for a discussion of the following problems: 

1. estimating and considering the cost of the non-associ- 
ative operations of the program. 

2. defining an “overall” cost function of space and time. 

3. considering the interaction between representation 
choices for separate data structures whose life-times 
overlap. 


RESULTS FOR THE SIMPLE EXAMPLE (Figure 2) 

Based on its analysis and on the answers to questions, the 
system proposes and evaluates several possible storage 
structures for the Sexof relation. The best of these is a 
record having one field which points to the set of MALES 
(this set is represented as a linked list). Based on the cost 
formulas, this has an expected size of 5.5 cells and an ex¬ 
pected time of 457 instructions. Another alternative would 
be to list each association separately in a table. This has an 
expected size of 13.0 and an expected time of 935. The set 
representation is better than this (and all others) in both time 
and space, and is chosen. This is a very simplified (almost 
degenerate) case of the selection system’s operation. 

In selecting a storage representation for the class based 
on the Parentof relation, the system goes through a similar 
procedure. In this case, it selects to assign a record for each 
person, with a field of the record indicating the set of par¬ 
ents. 

Now let us consider the more complex program including 
the statement in the box. Since the Homeof relation is used, 
there are now three classes of associations. In our tiny 
sample program, the only use of Homeof is to test if some¬ 
one lives in Rochester. This, plus the fact that there is a 
small fixed number of people, causes the selection system 
to choose as the best structure a single word per person 
indicating whether he lives in Rochester. This representation 
is estimated to require one cell and 300 instructions. 

A more interesting effect of adding the boxed statement 
is to change the choice for representing the class based on 
the Parentof relation. Now the program calls for enumer¬ 
ating all Child-Parent pairs as well as finding the PARENT 
of a given child. The selection system considers making two 
separate data structures or making one data structure which 
does both jobs. The best choice is to have a list of all 


children with each child indicating the set of his parents. 
This is estimated to require 16.5 cells and 1600 instructions. 

SUMMARY 

This paper explores some of the problems of automatic 
representation selection for associative data structures. In 
particular, it presents methods for selecting in-core storage 
structures for programs which use associative data and op¬ 
erations which are similar to the ones that the SAIL pro¬ 
gramming language provides. 

A model of the associative data and operations of such a 
program is defined. We describe a library of storage struc¬ 
ture representations for three-part associations (triples) that 
includes records, property lists, inverted lists and hash ta¬ 
bles. We show how to use the model in conjunction with 
the library to find “efficient” candidate representations for 
the associative data of a given program. This choice of 
candidate representations is guided by heuristic rules that 
embody educated assumptions about computation cost 
tradeoffs. In addition to representations that provide a single 
method of access to associative data, we consider represen¬ 
tations that provide multiple access paths to the data and 
ones that provide multiple copies of the data. 

Finally, we discuss methods for estimating the computa¬ 
tion cost of each candidate representation, and for selecting 
the “best” one. 

ACKNOWLEDGMENTS 

The author is indebted to Jim Low, Jerry Feldman, and Tom 
Cheatham for their ideas, guidance, and encouragement dur¬ 
ing the course of this work, and to Peggy Meeker, who put 
up with my drafts, changes and deadlines cheerfully while 
doing a meticulous job of editing and typing the manuscript. 

REFERENCES 

1. McCarthy, J. et al., LISP 1.5 Programmer’s Manual, M.I.T. Press, 1962. 

2. Feldman, J. A. and P. D. Rovner, “An Algol-Based Associative Lan¬ 
guage,” CACM, August 1969. 

3. Rovner, P. D. and J. A. Feldman, “The LEAP Language and Data Struc¬ 
ture,” 1F1P Congress Proceedings, 1968. 

4. Low, J. R., Automatic Coding: Choice of Data Structures, ISR16, Birk- 
hauser-Verlag, 1976. 

5. Maurer, W. D. and T. G. Lewis, “Hash Table Methods,” ACM Comput¬ 
ing Surveys, Vol. 7, No. 1, 1975. 

6. Rovner, P. D., “Automatic Representation Selection for Associative Data 
Structures,” Ph.D. Thesis, Harvard University; also TR 10, Computer 
Science Department, University of Rochester, 1976. 

7. Reboh, R. and E. Sacerdoti, “A Preliminary QLISP Manual,” SRI AI 
Center Technical Note 81, August 1973. 





Efficiency estimation—Controlling 
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People should be able to specify a program in a high level 
language and let the computer take care of finding an effi¬ 
cient target language implementation. One way to do this is 
by program synthesis. LIBRA is a system designed to guide 
the application of synthesis rules in the PSI program syn¬ 
thesis system. In the PSI system, there is a structured pro¬ 
gramming style of organization of programming knowledge. 
High level concepts such as sets may go through an arbitrary 
number of levels of refinement before target language con¬ 
cepts such as lists or arrays are produced. Refinement rules 
make the transformations between these concepts. If given 
such a set of refinement rules, LIBRA can guide the trans¬ 
formation of a high level program specification to a target 
language program. The target program is made efficient in 
two ways: by clever choice of refinements and by the ap¬ 
plication of optimizing transformations at any levels that are 
appropriate. 

LIBRA uses the technique of efficiency estimation to 
guide a heuristic tree search of some of the possible refine¬ 


ment sequences. Efficiency estimation takes into consider¬ 
ation: a user-specified program cost evaluation function, 
resources allowed for program writing, data structure sizes, 
and conditional branching probabilities. An estimate of the 
cost of the program descriptions in the tree is maintained 
for comparison between alternatives, for identification of 
the most important decisions, and to trade off final program 
performance with the cost of writing the program. To sim¬ 
plify the decision making process, decisions are grouped 
into related sets or cost-independent blocks. Heuristic 
knowledge about efficient implementations for certain cir¬ 
cumstances is also integrated into the system. A version of 
LIBRA has been written in INTERLISP; it has been used 
to guide the implementation of several LISP programs. 

This work is described in more detail in The Selection of 
Efficient Implementations for a High-Level Language in 
Proceedings of the Symposium on Artificial Intelligence and 
Programming Languages, Rochester, New York, August 
1977. 
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Transformational implementation 


by DAVID WILE and ROBERT BALZER 

Information Sciences Institute 
Marina del Rey, California 


Only modest gains in programming productivity have been 
produced in 25 years of software research, but the ground¬ 
work has been laid for major advances through rationaliza¬ 
tion and automated aids. This groundwork rests on two 
critical ideas: that specification must be separated from im¬ 
plementation, and that the separation between these two 
processes should be a formal operational abstract (i.e., very 
high level) program rather than a nonoperational require¬ 
ments specification. Structured programming represents the 
first results of combining these ideas. It is a special case of 
a more general two-phase process, called Abstract Program¬ 
ming, in which an informal and imprecise specification is 
transformed into a formal abstract operational program, 
which is then transformed into a concrete (i.e., detailed low- 
level) program by optimization. Abstract Programming thus 
consists of a specification phase and an implementation (op¬ 
timization) phase which share a formal abstract operational 
program as their common interface. 

The concept of Abstract Programming is completed by 
adding the feedback loops required by Testing, Mainte¬ 
nance, and Tuning. In conventional programming where no 
abstract program exists, these feedback loops all operate on 
the optimized concrete program. On the other hand, in Ab¬ 
stract Programming, if an effective method can be found for 
guaranteeing the validity of an implementation (that is, the 
functional equivalence of the abstract and concrete pro¬ 
grams), then the validation process can be shifted to the 
specification phase to show equivalence between the user 
requirements and the abstract program. Thus, validation 
could, and should, occur before any implementation. Fur¬ 
thermore, if the implementation process could be made in¬ 
expensive through computer aids, then maintenance could 
be performed by modifying the specification and reimple¬ 
menting it rather than directly modifying the optimized con¬ 
crete program, as is current practice. The importance of 
such an advance can be recognized when one realizes that 
optimization is the process of maximally spreading infor¬ 
mation (to remove redundant processing), and that modifi¬ 
cation requires information localization. Thus, the two pro¬ 
cesses are diametrically opposed; this fact explains much of 
the current problem with modifying and maintaining existing 
programs. The second major cause of this dilemma is that 


optimization obscures clarity and thus makes it difficult for 
maintainers even to understand how the concrete program 
operates. 

It is therefore clear that major advances in programming 
will hinge on the ability to provide an inexpensive optimi¬ 
zation process with guaranteed validity so that maintenance 
and validation can occur in the specification phase on the 
abstract program rather than in the implementation phase 
on the concrete program. 

Several researchers have espoused an approach which we 
will call Transformational Implementation (TI), in which 
equivalence-preserving transformations are successively ap¬ 
plied to the abstract program to effect such an optimization. 
The key to this approach is that while optimizing transfor¬ 
mations are selected by the programmer, the program is 
transformed by the computer system. Hence, the program¬ 
mer designs the optimization; the machine ensures its valid¬ 
ity and transforms the program. 

In addition to guaranteeing the validity of the implemen¬ 
tation, drastically reducing the time and effort required to 
implement a system, and enabling maintenance to be per¬ 
formed at the conceptual level on the abstract program, this 
approach would also allow programmers to experiment with 
alternative implementations to widen their experience base 
and improve their capacity to design optimizations. 

We have undertaken a study to discover: what facilities 
are necessary for the TI approach; what transformations are 
required; how are they expressed; what categories do they 
fall into; how can they be named; what kinds of applicability 
criteria are required; what processes are required to verify 
that these criteria are satisfied; what instrumentation facili¬ 
ties are required to test a transformation’s selection criteria; 
what proof techniques are required to validate the transfor¬ 
mation; etc. In short, the study will provide an experience 
base for the TI approach from which we can design and 
implement a prototype system. 

REFERENCE 

1. Balzer, Robert, Neil Goldman, and David Wile, “On the Transformational 
Implementation Approach to Programming,” 2nd International Confer¬ 
ence on Software Engineering, October 1976, p. 337. 
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A panel session—Whither automatic programming 


SESSION CHAIRMAN—ROBERT BALZER 

Information Sciences Institute 


Panel Members 

Thomas S. Standish—University of California 

Michael Hammer—MIT Laboratory for Computer Science 

PANEL OVERVIEW—Robert Balzer 

The preceding sessions have displayed some current re¬ 
search into the area of Automatic Programming. These ef¬ 
forts are small laboratory prototypes which attempt to au¬ 
tomate particular portions of the programming process. It is 
quite clear that to progress significantly beyond our current 
state of software development technology, increased auto¬ 
mation, in some form, must occur. 

The questions facing this panel, and addressed below, are 
when, in what form, and to what extent this automation will 
occur? 


THE FUTURE OF AUTOMATIC PROGRAMMING— 

Thomas A. Standish 

In automatic programming, we strive to build systems 
that will perform problem acquisition, algorithm and data 
representation synthesis, optimization, and concrete pro¬ 
gram generation—systems which automatically acquire 
specifications of the program behavior required and which 
will build programs to satisfy them. 

I speculate that the attainment of technological capability 
to build such automatic programming systems flexibly 
across a spectrum of dissimilar, important problem domains 
is a long way off—perhaps centuries and most likely dec¬ 
ades. Before we succeed, we shall likely have had to develop 
deep understanding of the interrelation between knowledge 
systems of many sorts—knowledge systems in which prob¬ 
lems are posed and in whose terms solutions must be deliv¬ 
ered (the so-called problem domains), knowledge systems 
describing the media in which problem solving computations 
must execute at the lowest levels (the so-called concrete 
solution domains), and knowledge systems that cooperate 
together to provide the scaffolding in which program syn¬ 
thesis can take place (the intermediate problem-solving 
knowledge systems, which offer systems of reasoning and 
representation). 


The exploration required to develop an understanding of 
relationships between these several sorts of knowledge sys¬ 
tems is likely to lead to a deep reorganization of the foun¬ 
dations of epistemology itself. For example, the doctrine of 
reductionism —which holds that phenomena at higher levels 
of science can be explained by reduction to laws and facts 
at lower levels—is a doctrine which is far too simplistic and 
inadequate to provide understanding of how to build pro¬ 
gram synthesizing systems. Reductionism may describe the 
end product of program synthesis—i.e., the already synthe¬ 
sized program—but it seems inadequate to describe the 
process of synthesis itself. 

The doctrine of reductionism is important in program¬ 
ming. Every large software system is living testimony to the 
idea of reductionism. In such systems, we organize the data 
and operations at given levels to represent the information 
and problem solving processes required at higher levels. 
When we engage in “top-down programming” or “program¬ 
ming by stepwise refinement,” we consciously organize the 
act of large system construction into a number of levels of 
reductionistic choice, each carefully selected to be intellec¬ 
tually manageable and to permit flexibility in future main¬ 
tenance. 

While every synthesized software system is an example 
of successful reductionism in which the external high-level 
behaviors of the system have been reduced to a composition 
of the lowest level behaviors of the underlying machine, 
there may be a paradox. The knowledge systems in the 
heads of programmers who build the overall system may not 
be similarly reducible. Rather, such knowledge systems may 
exhibit phenomena akin to the logical independence of the 
parallel postulate from the remaining axioms of Euclidean 
geometry. 

The history of science teaches us that large increments in 
technological power are seldom purchased through cheap 
tricks or shallow understanding. Rather, deep basic under¬ 
standing of a degree of indirectness scarcely imagined by 
original explorers seems more characteristic (recall here the 
history of the conquest of flight, or the history of the quest 
to transmute the elements). 

It seems to me that basic understanding of how to repre¬ 
sent knowledge systems, and how to use one knowledge 
system to represent and solve problems posed in another, 
is a basic requirement of the task of building program syn¬ 
thesis systems. Perhaps it is better to make an indirect assault 
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on deepening our knowledge in these areas before making 
direct assaults on the synthesis of particular categories of 
programs. After all, the Wright Brothers built a wind tunnel 
and studied properties of airfoils extensively before they 
conquered flight. 


THE IMPACT OF AUTOMATIC PROGRAMMING 

RESEARCH—Michael Hammer 

Contemporary automatic programming research encom¬ 
passes a variety of approaches to the goal of transferring 
some of the programming function from human to machine 
and thereby alleviating the software problem. A very high 
level language system translates a program expressed in 
terms of nonprocedural, problem-oriented constructs into 
conventional algorithmic code; in so doing, it must perform 
global program transformations in order to achieve an effi¬ 
cient object program. A knowledge-based system uses the 
semantics of a particular problem domain to interpret a 
user’s functional specifications for a program in that domain 
and to construct a system with the specified functionality. 
A program synthesis system utilizes formal theorem-proving 
techniques to transform non-functional program specifica¬ 
tions (typically expressed in terms of a set of input-output 
pairs) into an executable program. 

Although extensive research has been conducted in each 
of these areas, and significant advances have been made in 
addressing the technical problems that each presents, it is 
unlikely that any of this work will culminate in systems that 
will meet the ambitious goals that have been set for them. 
The very high level languages that have been developed so 
far are adequate for expressing the main body of a program 
in a nonprocedural way; however, they do not address issues 
related to deviations from the normal flow of the computa¬ 
tion, such as special case recognition, exception handling, 
and coping with erroneous data or system failure. Further¬ 
more, introducing a new language into an operational envi¬ 
ronment has in the past proven to be a major undertaking, 
one that raises serious managerial and technical problems. 
The principal problem facing the more advanced, artificial 


intelligence systems is one of scale. It is doubtful that the 
techniques that suffice for laboratory prototypes can be ex¬ 
tended to cope with realistically rich problem domains and 
large, complex programs. 

The difficulty underlying all of these problems is that the 
fundamental premise of automatic programming is simply 
false. This premise is that, after thirty years’ experience in 
writing programs, we now know enough about the program¬ 
ming process to automate it. On the contrary, it appears that 
our understanding of the nature of programming is weak. 
The ad hoc attacks on the software problem that are cur¬ 
rently feasible are not likely to produce major results. 

Nonetheless, automatic programming research will have 
important consequences, both direct and indirect, for the 
future of software construction. In the near term, we should 
see limited systems that automate one part of the program¬ 
ming process or that are adequate for the synthesis of small 
programs; in either ease, the context will be a simple prob¬ 
lem domain that has a well-defined and understood semantic 
structure. Examples of such systems include an automatic 
data base design system and a facility for generating simu¬ 
lation studies. Another possibility is a system that uses 
knowledge of an application domain to assist a user in de¬ 
riving a complete and consistent set of program specifica¬ 
tions. In another vein, very high level language research 
should lead to the design of formal specification languages 
for effective and precise inter-personal communication. 

However, the most significant benefit of automatic pro¬ 
gramming research should be a deeper understanding of 
programming by human programmers. It has often been the 
case that the attempt to automate an activity has forced its 
practitioners to understand and systematize their actions; 
automatic programming research should have the same ef¬ 
fect on programming. In the attempt to computerize the 
software production process, it is uncovering important gen¬ 
eral programming principles and techniques. The structures 
and guidelines that derive from this work will enable and 
encourage programmers to be more disciplined and organ¬ 
ized; in the long run, they can contribute to turning pro¬ 
gramming from mysticism into science. In other words, au¬ 
tomatic programming research should benefit software 
engineering practice. 




PART in—SYSTEMS 




Area Co-Director: 
Norman Abramson 
University of Hawaii 
Honolulu, Hawaii 


Area Co-Director 
Eugene R. Cacciamani, Jr. 
American Satellite Corporation 
Germantown, Maryland 


Data networks 


In this area we explore the technology and development of information net¬ 
works on a national and international basis. There are four sessions in this area 
and they deal with (1) “The Architecture and Application of Nationwide Packet 
Switching Networks,” (2) “International Computer Communications,” (3) “In¬ 
ternational Public Data Networks,” and, (4) “Satellite Data Communications for 
Public Service.” 

The first session focuses upon the architectural similarities and differences of 
several packet-type networks and how the application of these networks influ¬ 
ences network architecture which have recently become operational or are under 
development. 

The first of two sessions on international computer networks will cover match¬ 
ing future international regulations against the continued growth of computer 
communications systems and technology utilizing all types of international trans¬ 
mission media. The concerns of this session include the non-homogeneous nature 
of national regulations as they apply to international computer communications 
systems, transborder data flow, and the need for new regulatory considerations 
for international satellite-oriented computer communications networks 4 

The second international session emphasizes the operational problems of in¬ 
ternational data networks with multiple organizations interconnecting dissimilar 
equipment rather than a single, coordinated, homogeneous network. This inho¬ 
mogeneity creates special problems and requirements in order to create a viable 
service. This session will describe some aspects of the interconnection problem, 
potential solutions, and some internetworking field trials underway. 

The final session will deal with the emerging question of information networks 
for public service users in the United States. These users consist of a small 
number of related communities—libraries, educational institutions, health cen¬ 
ters, and local government agencies. Each of these communities has a need for 
data communication capabilities with certain common characteristics. Among 
these characteristics are a high degree of connectivity within a highly dispersed 
user community, the ability to distribute relatively small amounts of data with 
high economic or social value and a modular structure which can accommodate 
rapidly changing and diverse data rates within the same system. The key to 
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satisfying these data communication needs is the aggregation of the public service 
data communications market by means of communications satellites and the 
broadcast data communications architecture they make possible. In this session 
we describe the projected requirements of this aggregated market and discuss the 
status of current proposals to provide these data services. 



Challenges in the planning of international communications 


by DAVID J. HORTON 

Hawaiian Telephone Company 
Honolulu, Hawaii 


It’s difficult sometimes to realize how far international tel¬ 
ecommunication has come in 20 short years. As recently as 
1957, “Hawaii Calls” was broadcast using HF radio. After 
the program was switched to transpacific cable, simulated 
fade had to be added to recapture the flavor of “music from 
the isles.” Today we’ve progressed to the point where we 
can provide a small store on Molokai with on-line data 
communications to a data base in Paris—at low cost and 
virtually error free. 

I’d like to explore with you the environment under which 
all this achievement came about and to examine some real 
areas of concern which may inhibit continued progress. 
While some of my remarks may apply generally, my em¬ 
phasis will be on data communications in the Pacific Basin. 

I have a very simple definition of planning: A process 
which results in spending the right amount of money, on the 
right amount and right kind of equipment at the right time. 
The definition is simple; the process, alas, is always difficult 
and rarely perfect. Planning in the face of rapid technological 
change and everchanging market needs is particularly tricky. 
When the international dimension is added, the complexity 
reaches quite awesome proportions. Yet, decisions must be 
made or opportunities will be lost and/or needed services 
will not be provided. 

Let’s look at some of the elements that enter the picture 
when we go international: 

— Apart from the obvious problems of language and cul¬ 
tural differences, we must recognize the different ways 
that telecommunication is viewed in different coun¬ 
tries. In North America for example, we aim for the 
lowest possible cost consistent with good service and 
a fair rate of return. Most other countries are inclined 
to view telecommunication as a source of high profit 
with which to subsidize some of the weaker opera¬ 
tions, most notably the postal service. 

— Political considerations cannot be ignored. Here we 
must be concerned not only with political ambition, 
but with the opportunities of political progress. We 
can contemplate, for example, the phenomenal in¬ 
crease in communications which would accrue from 
full trade with the People’s Republic of China. 

— We must concern ourselves quite deeply with some of 
the reservations being expressed regarding the pres¬ 
ence of data bases in foreign countries. Not only is 


this viewed as an unnecessary import, but there is very 
valid worry regarding what is left of our privacy. 

— Markets are different as we travel around the world. 
We find different terminal types, different protocols, 
and different speeds—happily, however, major suc¬ 
cess is being achieved in the area of international 
standards. I can think of no area more deserving of 
our continued attention than this. We, as carriers, have 
a profound obligation to ensure a high degree of uni¬ 
formity in the networks we design for Spaceship Earth. 
We have the skills, and we have the vehicle (CCITT)— 
we will have no excuses for failure. 

— Geography differs widely, as is obvious when one 
compares the wide open spaces of Canada with the 
density of population in the Netherlands. 

— In satisfying customer needs with new equipment, it 
is frequently not enough that it was invented “here”, 
but it also must be designed here, developed here, 
manufactured here, sold here, and serviced here. And, 
often the customer must wait until these national ob¬ 
jectives can be satisfied. 

— And finally, planners must contend with all of the 
unpredictability inherent in the presence of extensive 
competition for international telecommunications. 

But, complex as it may be, planning must continue. Good 
planning is essential and plans, once agreed, should meet 
time and cost targets or both carriers and customers will 
suffer. This is particularly true of international planning 
where the inability of planning members to fulfill their com¬ 
mitments can cause considerable difficulty for their corre¬ 
spondents. 

International planning in telecommunications has had its 
problems to date but has nothing to be ashamed of. It is 
interesting to recall that the first major achievement was the 
laying of the telegraph cable between two countries which 
today provide the butt of many a joke; namely, Newfound¬ 
land and Ireland—and that this occurred in the year 1856, 
20 years before the invention of the telephone. 

Progress since then has been quite rapid (with most of it 
coming within the last 15-20 years) to the point where we 
now have extensive cable and satellite facilities encircling 
the globe. 

At the 6th Annual International Conference on Planning, 
held last fall in Honolulu, we were treated to paper after 
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paper about the growth in the Pacific Basin with special 
emphasis on the emergence of Korea, Taiwan, Singapore, 
and Hong Kong as “new Japans” with all that it implies in 
terms of increased trade and thus, telecommunication 
traffic. It is obvious that good planning is essential if we are 
to meet the emerging needs of customers in these Pacific 
Rim countries. 

Thus far, we’ve gone through a progression of complexity: 
fast moving technology; increased and changing market de¬ 
mands; and a high requirement for international cooperation. 
We must now add yet another challenge—international data 
communications. 

The past few years have seen explosive growth in inter¬ 
national data communications and with the advent of packet 
switching, this growth will continue at an ever-increasing 
pace. This latest technological wonder which has been with 
us a very short time finds its roots in the ARPA network 
but its branches now encircle the world. At last count, there 
were almost 20 such networks in service or about to be in 
service. 

Packet switching respects no boundaries. It was interna¬ 
tionally almost before it became domestically available. The 
Atlantic was the first pond to be hurdled, and we have today 
extensive access to North American data bases by European 
customers. Similar plans are already well under way in the 
Pacific area. 

Technologically, packet switching is a hyperactive child, 
constantly exploring new possibilities—and now and again 
getting itself in a little trouble. The pace of technological 
change is higher than in any other area of telecommunica¬ 
tions and with the introduction of microprocessors, we find 
ourselves using third generation technology within three 
years of the first network going into commercial service. 

Data communications customers are a demanding breed— 
demanding that new technology be introduced fast, de¬ 
manding shorter time frames than they have traditionally 
been promised, demanding the features that they need, and 
perhaps most importantly, attaching a very valid threat of 
taking their business elsewhere (or doing it themselves) if 
these demands are not met. 

There is of course much to be done, and I’ve touched on 
some of the tasks: 

• harnessing the right technology to meet real market, 
needs 

• standards 

• international dialogue 

One area, however, which could become a major stumbling 
block is not within the carriers’ control. I’m speaking, of 
! course, of regulation, or more precisely, the regulatory proc¬ 
ess. 

Let me suggest that regulation is here to stay and that it 
will probably get worse before it gets better. 

In a December 1977 Time magazine. President Carter was 
quoted as follows: “Regulations should be as simple and 
clear as possible. They should achieve legislative goals ef¬ 
fectively and efficiently. They should not impose unneces¬ 


sary burdens on the economy, on individuals, on public or 
private organizations, or on state and local governments.” 

How about that? That was in December 1977. You all 
notice the big improvement? 

To be fair, the problems are not entirely the fault of the 
FCC. Once the decision is made to have controlled com¬ 
petition as opposed to a monopoly, it follows logically that 
there will be a greater workload. It is the FCC’s inability to 
handle this workload that causes so much pain, not only 
domestically, but also with our partners in other lands. At 
a meeting of Pacific cable operators held in Honolulu at the 
Kahala Hilton Hotel in September 1977, all countries ex¬ 
pressed dismay at the regulatory lag caused by the FCC. In 
one cited example, a submarine cable cut into service one 
and one-half years late and cost $40 million over budget as 
a result of this delay. 

And, if I may use an example much closer to my own 
experience, let me tell you that from the start of the initial 
discussion with Telenet until the installation of a working 
node in Honolulu, less than three months expired. More 
specifically, from the placing of the order to the acceptance 
of installation was a mere six weeks, or more specifically, 
still, from the delivery of equipment to turn up (capable of 
providing service) five working days expired. By compari¬ 
son, our request for approval to construct was filed on 
August 12th, 1977 and approved about December 15th with 
an effective date of January 3rd, 1978. I think you can 
imagine the disappointment that is caused to carrier and 
customer alike as a willing switch capable of earning revenue 
sits gathering dust as the paper flows around Washington. 
This is simply not adequate regardless of what the problems 
and what the excuses are. 

The fact is that whether we like it or not (and I personally 
do) technology is moving fast and market demands are in¬ 
creasing daily, resulting in the need for fast response from 
carriers both domestically and internationally. The regula¬ 
tory process must keep up. 

How do we do this? I wish I knew. 

I don’t believe that simply increasing the staff at the FCC 
is the answer. So let me offer some suggestions which may 
be of help: 

1. Establish guidelines and policies for international 
telecommunication. 

2. Clarify the roles of individual carriers if in fact there 
are any valid reasons for segmenting the market among 
them. 

3. Anticipate the effect of technological likelihoods, and 

4. Based on carrier-provided information, anticipate the 
kind of action that market demands will require not 
only of the carrier, but of the regulatory process itself. 

Let me give you an example: 

We’re now in the middle of an extremely costly exercise 
called Computer Inquiry II. There’s a great deal at stake 
and some very vocal and widely different views are being 
expressed by various lobbies. What bothers me most about 
this is that it’s only six years since we were presented with 
the wisdom we have come to know and love as the computer 
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inquiry. But it wasn’t the computer inquiry—it lasted a mere 
six years. Why? Mainly because its authors failed to under¬ 
stand where technology and the demands of the market 
place were taking us. And at the very moment in history 
when everyone finally realized the line between communi¬ 
cations and data processing was a broad blur, the FCC 
presents us with a definition which claims to draw a major 
distinction. It’s thus difficult to be optimistic that CI2 will 
be a big improvement on its predecessor. 


Let me stress again that we must have a thorough under¬ 
standing of what technology can do, what the customer 
wants, and what the various vendors are trying to achieve. 
We must then produce regulations that are fair, of course, 
but which stimulate innovation and keep prices as low as 
possible. A difficult task, indeed, but one which must be 
undertaken. I believe the carriers themselves should play an 
active role in trying to streamline regulation. I would hope 
that both they and the FCC would welcome this. 





The need for continuation of full-period 
transparent private-line service 


by PHILLIP C. ONSTAD 

Data Services, Control Data Corporation 
Greenwich, Connecticut 


During the past year several moves have been made by 
the international telecommunications industry which may 
threaten the continuation of the flexibility currently available 
to the user of telecommunications services for data trans¬ 
mission. The major problem now confronting the user of 
international telecommunications services for data transmis¬ 
sion is the hinted or announced intention of the international 
and foreign carriers to offer “value added” services as a 
replacement rather than an alternative to full-period trans¬ 
parent private-line services. The availability of transparent 
private circuits permits technological innovation in signaling 
and in information processing network development to the 
benefit of all. Loss of transparent, flat-rate, private circuits 
would retard advances in distributed processing and shared 
data base developments. Furthermore, many existing tele¬ 
processing systems would be degraded in effectiveness and 
in many cases their data services would be withdrawn. 

With the advancement of computer technology during the 
fifties and sixties, it became evident that the full potential 
of data processing and information systems would not be 
forthcoming until a faster and more efficient means of trans¬ 
portation could be developed to move information in and 
out of the data processing systems. While data processing 
systems had the power to manipulate information at split 
second speeds and store vast amounts of data which could 
be retrieved in fractions of seconds, these capabilities were 
of little use unless the user was within close proximity to 
the data processing system. Transportation by means of mail 
or delivery services worked well for batch processing op¬ 
erations such as payroll, sales analysis, and other operations 
where there was no need for an immediate answer to a 
problem. However, these transportation methods could not 
satisfy the demands of applications such as reservation sys¬ 
tems or production control systems where information was 
needed on an immediate basis. 

The obvious solution to this problem lay in the use of the 
vast telecommunications networks covering the country and 
the world. Although these networks operated in an analog 
mode and were used for transmission of voice and low speed 
telegraph signals, equipment was developed for connection 
between the data processing systems and terminals and the 
telecommunications network which could convert the sig¬ 
nals from digital to analog and then back to digital at the 


other end. Thus began the era of telecommunications de¬ 
pendent data processing systems. 

In the early days of data transmission, the speed of trans¬ 
mission was limited by many factors and the reliability of 
the transmission was low. In addition, there was very little 
interest by the communications industry, whose major pur¬ 
pose was to provide voice and low speed message commu¬ 
nications service, and not to provide methods for the high 
speed data transmission and high reliability factors required 
by the data processing systems. Thus, it was left to the data 
processing and related industries to develop the necessary 
equipment and procedures. 

The telecommunications common carriers offer transpar¬ 
ent telecommunications circuits on both the private-line and 
switched networks; these are available in various speeds. 
By utilizing these circuits, the data processing industry de¬ 
veloped or perfected equipment and techniques which made 
the transparent communications facility a highly efficient 
and highly reliable method of transportation of information 
from remotely located input/output devices to and from the 
data processing system. 

Once this means of transportation was proved feasible, 
the number of telecommunications-oriented data processing 
applications exploded. This demanded increasing technolog¬ 
ical development to obtain optimum performance of the 
transportation medium in order to maintain optimum per¬ 
formance of the overall data processing system. 

The data processing industry has been and continues to 
be a highly competitive and highly complex industry with 
many companies manufacturing and marketing many differ¬ 
ent types of equipment and systems from the very small 
micro processors to huge information handling systems. To 
remain competitive in the marketplace each company has 
developed unique architecture, technologies, coding meth¬ 
ods, circuit logic, storage methods, protocols, etc., in order 
to make their systems more reliable and faster than others. 
Each system has a specific balance of hardware and software 
control to maintain optimum performance from a specific 
system and specific application. In maintaining this balance, 
the user must also have control over all input and output 
functions, including remote terminal devices that carry in¬ 
formation to and from the system. 

When preparing a new application for a data processing 
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system, the programmer and system designer must have the 
flexibility to utilize the best combination of computer logic, 
storage media, input/output methods and devices. Further 
design must be made to the transportation system from 
remote terminal devices to the processor in such a way that 
information is received and released in a manner which 
meets the requirements of the application, such as formats 
for messages, etc. It must also meet the requirements of the 
data processing system to properly mesh these input/output 
operations with the multitude of other operations going on 
during the processing of a major application. The importance 
of the input/output operations increases as the complexity 
of the application increases. The effect of several low speed 
terminals coming into a system or application which is pri¬ 
marily batch oriented is minimal and does not require a great 
deal of control over the input/output method. However, a 
system operating a large reservation or time sharing appli¬ 
cation involving hundreds of terminals requires a highly 
complex input/output protocol with almost total control over 
the telecommunications activity in order to maintain opti¬ 
mum system performance. 

The transparent nature of the telecommunications net¬ 
works has given the application designer the flexibility to 
design the transmission protocol and utilize equipment 
which best meets the requirements of the overall system 
environment. The designer may choose the speed of trans¬ 
mission, the line protocol to be used, the coding structure, 
block lengths, error correction methods, retransmission 
methods, etc. He has the flexibility to utilize the usage- 
sensitive switched circuits of the public networks or full- 
period line service. He also has the flexibility to utilize 
asynchronous or synchronous transmission methods and 
may utilize various types of equipment such as multiplexors, 
concentrators etc., to obtain maximum utilization from the 
telecommunications service. This flexibility is extremely im¬ 
portant to the operation of a major telecommunications de¬ 
pendent data processing application. Hundreds of data proc¬ 
essing systems of this type are operating throughout the 
world today. Many of these systems require thousands of 
miles of private lines which are leased from the common 
carriers, both nationally and internationally. They also con¬ 
nect to the public-swtiched network for many applications. 

Several years ago a new method of data transmission was 
introduced with the advent of the “value added” carriers. 
This group of regulated carriers lease full period private-line 
services from other carriers and, through the use of various 
types of telecommunications and data processing equipment 
and software in conjunction with the private line services, 
establish a stand alone data transmission network with a 
protocol best suited to the operation of the network. This 
“valued added” service then is offered to users who do not 
wish to establish and operate their own telecommunications 
networks but have some telecommunications requirements 
in their data processing applications. 

Although these “value added” data transmission services 
do fulfill the telecommunications network requirements of 
many users operating data processing systems with small- 
or medium-scale remote terminal operations and further 
allow the user to share a network with other users, in most 


cases they are of little value to the user with a large tele¬ 
communications network requirement. This is because they 
have been designed with the optimum performance of the 
network rather than the optimum performance of any single 
application as the criterion. In utilizing these networks, the 
user must conform to the protocol requirements of the 
“value added network,” therefore losing much of the flex¬ 
ibility available in the transparent network. This is not to 
say that the “value added networks” do not furnish a val¬ 
uable service to the small- and medium-scale user on a 
national basis. Indeed, the savings in telecommunications 
cost available through use of this type of service is in many 
cases such that they provide the economic feasibility of 
performing some telecommunications functions in an appli¬ 
cation normally run in batch mode. 

In the area of international data transmission, the value 
added services provide the only means of communications 
on a usage sensitive basis between the United States and 
foreign points. The only other alternatives available to the 
user are the very slow telex service at high cost or the use 
of full period international private-line service, which, be¬ 
cause of its extremely high cost, is only economically fea¬ 
sible in the cases of very large requirements or with users 
needing an alternative voice data capability which might 
offset the high cost. In many cases even the large user will 
find it more feasible to utilize the value added type service 
to points having small transmission requirements even 
though response and/or system optimization may suffer 
somewhat when adapting to the protocol of the value added 
service. 

To summarize, the intelligent networks provided by the 
value added services will fulfill the requirements of a great 
number of telecommunications-dependent data processing 
applications and systems. However, the large variety of data 
processing systems, each operating under a unique architec¬ 
ture and the various requirements of the large telecommu¬ 
nications dependent data processing applications, demand 
that the user have complete flexibility to utilize the best data 
transmission method possible. This is true whether those 
methods be transparent or value added service, private-line 
or switched network service, in order to design and operate 
the optimum environment for the data processing applica¬ 
tions. 

During the past year several moves have been made by 
the international telecommunications industry which may 
threaten the continuation of the flexibility currently available 
to the user of telecommunications services for data trans¬ 
mission. 

As previously stated, the new value added type services 
make available a viable alternative to fulfill the needs of a 
significant number of telecommunications users. However, 
these type services have serious drawbacks and cannot be 
applied to all user requirements. They are also relatively 
new and have yet to be totally proven from economic as 
well as reliability standpoints. While the growth of these 
services has been somewhat slow within the United States 
with only three independent carriers currently offering the 
service, the international common carriers, as well as a 
number of foreign telecommunications administrations are 
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moving ahead at a substantial pace to have these value 
added services in place as soon as possible. This has led to 
additional problems to the users as the fast pace toward 
installation of these services has not allowed the proper 
consideration for much technical standardization among the 
various services, thus creating a number of potential prob¬ 
lems for the user. 

The technical problems leading toward some type of 
standardization of these networks is currently being handled 
by the Study Group VII of the Consultive Committee on 
International Telegraph and Telephone of the International 
Telecommunications Union. Although this group has made 
significant progress toward solving of the standardization of 
the protocols and other technical criteria which will allow 
for the eventual interconnection of these networks, much 
work must still be done before proper solutions are found 
to make these networks truly viable on an international 
basis. 

The major problem now confronting the user of interna¬ 
tional telecommunications services for data transmission is 
the hinted or announced intention of the international and 
foreign carrier to offer these new services as a replacement 
rather than an alternative to full period transparent private 
line services. 

The possible replacement of full period transparent private 
line services for international and foreign communications 
has been brought to the attention of users during the past 
year by: 

• The introduction of a contribution by the Italian dele¬ 
gation to the CCITT Study Group III on Rates and 
Tariff Matters requesting the group to study the feasi¬ 
bility of replacing full-period private-line services with 
some method of usage-sensitive or volume-sensitive 
pricing for the service. 

• The refusal of an intemaional common carrier to furnish 
full-period private line service to the United States. The 
refusal is based on the availability of a new usage-sen¬ 
sitive service between that point and the United States 
which would now fulfill prospective customers’ require¬ 
ments. 

• Articles now appearing in the Japanese press which 
indicate that Japan plans to have their new Venus Value 
Added Service operating within the next year or so to 
replace for private line services for data transmission. 

Solutions to these problems which are of true benefit to 
the user are complex and difficult to obtain. With the ex¬ 
ception of the United States Federal Communications Com¬ 
mission, which is charged with the protection of the public 
interest in the area of telecommunications, there is very 
little opportunity to present a user’s point of view in inter¬ 
national and foreign telecommunications services. 

Since the only international body having an interest in this 
area is the Consultive Committee on International Telegraph 
and Telephone of the International Telecommunications 
Union, it is of critical importance that users take an active 
interest in the activities of the American delegation to the 
CCITT. 


In view of the pattern of rapid technological advancement 
and economic growth in the data processing and telecom¬ 
munications industries, we believe that a study of the tariff 
system for full period private line service may be well ad¬ 
vised. Because the tariff system profoundly affects the uti¬ 
lization and development of communications based data 
processing systems as well as other communications based 
systems, we believe that such a study, if properly handled 
and provided it receives input from the user community as 
well as the carriers, may result in workable solutions to the 
issues at hand. However, such a study must be conducted 
with a very open mind and without preconceived notions on 
the part of the international and foreign carriers that replace¬ 
ment of full period transparent private line services is the only 
solution. 

There can be no doubt that the world has entered the 
information age where continued growth of all elements of 
society are mutually dependent on the rapid flow of infor¬ 
mation with the minimum of hindrances. The EDP industry 
has spearheaded the advancement of information transfer 
through its technological and manufacturing advances. 
Prominent among those applying these advances to the di¬ 
rect benefit of the public are the financial, retail and health 
care activities and certainly the telecommunications admin¬ 
istrations. 

A key factor in the technological advance of ever-increas¬ 
ing telecommunications oriented data processing has been 
the widespread availability of full-period point-to-point and 
multipoint private leased circuits. The most important char¬ 
acteristics of privately leased circuits have been their trans¬ 
parency and a tariff system independent of the volume of 
information transferrred. Transparency as used in this con¬ 
text denotes a point-to-point or multipoint analog circuit 
which delivers the electrical signals applied by the user with 
the practical minimum of delay and distortion consistent with 
the applicable tariffed characteristics. 

The availability of transparent private circuits permits 
technological innovation in signaling and in information 
processing network development to the benefit of all con¬ 
cerned. Presently, these innovations can proceed observing 
only the electrical interface without the constraints of all 
establishments and telecommunications network administra¬ 
tive control protocols and without the potentially excessive 
delays even in virtual circuits. 

There is no doubt that the introduction by the administra¬ 
tions of new services will stimulate the growth of new in¬ 
formation systems which would not have emerged other¬ 
wise. There is no essential conflict between the new services 
and privately leased circuits and it would be contrary to 
every interest for any tariff system to be based on an un¬ 
founded fear of competition. 

Loss of transparent flat-rate private circuits would indis¬ 
putably retard advances in distributed processing and shared 
data base developments. Furthermore, many existing tele¬ 
processing systems would be degraded in effectiveness and 
in many cases their services would be withdrawn. Addition¬ 
ally, and as a direct consequence, administrations would 
suffer erosion of their revenues rather than safeguarding 
their revenue. 




Satellite business systems—Innovative services for business 
communications 

by RONALD W. McCABE 
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INTRODUCTION 


It is a pleasure to participate in the National Computer 
Conference, and to represent SBS in this session on Inter¬ 
national Computer Communications. The topic is a most 
timely one for SBS, even though we are a domestic carrier, 
because during our extensive market investigations the de¬ 
sire of potential customers to improve their ability to con¬ 
nect their domestic and foreign data transmission capabili¬ 
ties has been expressed quite strongly. SBS believes its 
proposed offering will provide customers a significantly im¬ 
proved digital network capability within the United States, 
and SBS is supportive of industry efforts such as this which 
recognize these expressed concerns of users and which seek 
to promote an improved environment for uniform interna¬ 
tional networks. But before I discuss the specific topic of 
today’s meeting, I would like to give you some background 
on SBS and describe its plans and recent activities. 

SBS proposes to offer innovative, high data rate, private 
switched communications networks for industrial and gov¬ 
ernmental users. These networks will be designed primarily 
for organizations requiring large communications networks 
that have heavy and dynamic traffic loads, and will provide 
integrated digital transmission of voice, data, and image 
traffic among geographically dispersed locations—much as 
a private branch exchange (PBX) provides switched com¬ 
munications within a single facility. Importantly, it is this 
same class of user which is more likely to be multinational 
in scope and to have far-reaching requirements for various 
types of international communications. Thus, SBS’ market 
emphasis is closely allied with the matters of interest here 
today. 

As you are probably aware, the SBS system will operate 
at higher frequencies (12/14 GHz) than present terrestrial 
and satellite carriers to minimize frequency interference with 
other systems. This will permit more geographical flexibility 
in the location of earth stations than is possible at the lower 
frequencies. These earth stations, which will incorporate 
specially-developed modulation and access equipment, will 
be located on the customer’s premises, thereby minimizing 
the cost of access from customer nonearth-station locations 
to the nearest earth station customer location. 


Satellite Business Systems is a partnership of COMSAT 
General Business Communications, Inc., Information Sat¬ 
ellite Corporation, and Aetna Satellite Communications, 
Inc. These partners are, respectively, wholly-owned subsid¬ 
iaries of COMSAT General Corporation, International 
Business Machines Corporation, and Aetna Life & Casualty 
Company. 

SBS filed applications with the Federal Communications 
Commission in December 1975 and obtained approval for 
operation in February 1977. Major system developments and 
procurements are now under way, and we look forward to 
implementation. SBS expects to begin providing services 
with its operational system in early 1981. Prior to that time, 
SBS will be engaged in an intensive pre-operational program 
using leased satellite and terrestrial transmission capacity to 
connect a number of IBM locations; this activity is designed 
to evaluate the service approach and communications tech¬ 
niques planned in the operational system. The first phase of 
this effort, now successfully completed, involved experi¬ 
mental traffic tests between SBS earth stations at Pough¬ 
keepsie, New York and Los Gatos, California. Phase II has 
been under way since early this year and consists of a com¬ 
mon carrier service, offered under terms of a tariff on file 
with the FCC, among these two locations and a third SBS 
earth station installed in Raleigh, North Carolina. An im¬ 
portant objective of the Phase II program is to provide a 
stable environment into which components of the SBS op¬ 
erational system can be introduced and field trials con¬ 
ducted. 

Another activity designed to demonstrate and better un¬ 
derstand the usefulness of enhanced communications in the 
business environment was the PROJECT PRELUDE ex¬ 
periment which was recently completed. SBS was author¬ 
ized by the FCC and NASA to use the Communications 
Technology Satellite, owned jointly by NASA and the Ca¬ 
nadian Department of Communications, and transportable 
earth stations to conduct an innovative business communi¬ 
cations experiment with newly-developed terminal equip¬ 
ment and business machines. The experiments were con¬ 
ducted successively at three pairs of business locations at 
the host companies—Rockwell International, Texaco, and 
Montgomery Ward. These companies were selected because 
they represented a cross section of the user community, and 



722 


National Computer Conference, 1978 


helped to assure that representative data were collected. 
Experimental transmission of high speed data, high speed 
facsimile, and voice and televisual communications took 
place and the results were reported in the spring to both 
NASA and the FCC. In general, these experiments solidified 
our belief that these emerging applications will indeed satisfy 
unmet communications demands and become widespread in 
future networks. 


SBS UNIQUE CHARACTERISTICS 

Now let me describe briefly the uniqueness of the services 
that SBS plans by contrasting them with present communi¬ 
cations. Satellite services are generally provided through 
very large earth stations located at some distance from urban 
centers because of their size, and to avoid interfering with 
terrestrial microwave facilities. Some small earth stations 
have been introduced but only in a limited way, again to 
avoid interfering with existing systems. The services cur¬ 
rently offered by both terrestrial and satellite systems are 
primarily of an analog, full-time, point-to-point nature, and 
the switched services currently available are for voice and 
low-speed data. 

In contrast to this environment, SBS will offer innovative, 
high data rate, private switched communications networks. 
These networks will provide integrated digital transmission 
of voice, data and image traffic among geographically dis¬ 
persed locations, with full switching capability from one 
location to another. 

Network is a key word in understanding SBS’s unique¬ 
ness. It implies that there is connectivity among all of the 
locations at any time. This particular feature is perhaps the 
primary factor in SBS’s system design. SBS will provide a 
customer complete connectivity among all his earth stations 
with sufficient transmission capacity to meet normal oper¬ 
ating requirements; additional capacity will be available, on 
demand, to meet incremental requirements. This service 
approach, offered through small earth stations located on 
the customer’s premises, results in a single multi-application 
network capable of handling virtually all of a customer’s 
internal communications. 

Locating earth stations on the customer’s premises also 
provides a high level of security. This, combined with the 
use of time division, multiple access (TDMA), will make it 
impractical to intercept or decipher communications. The 
modulation and access equipment, burst modem, and radio 
frequency equipment of the earth station operate in a con¬ 
trolled access environment. Further, privacy equipment 
(such as cryptographic encoders) can be adapted for use as 
required. 

SYSTEM COMPONENTS AND MANAGEMENT 

To provide these dynamic, switched high data rate digital 
transmission services to the locations of an enterprise, SBS 
will install at each customer’s traffic concentration points 


earth stations, owned and operated by SBS, through which 
he will obtain access to his network. The stations will use 
antennas five or seven meters (16 or 23 feet) in diameter and 
will include innovative modulation and access equipment to 
accomplish switching and multiplexing and to control the 
assignment of satellite capacity among the earth stations 
serving each customer. A variety of access ports is provided 
in each earth station for analog signals and for a broad 
spectrum of digital rates, based on the amount and types of 
traffic requirements of that particular station. 

The system that SBS intends to implement initially will 
include three satellites—two in orbit (one spare), and one 
ground spare. As part of the SBS service, the customer will 
be able to obtain satellite transmission capacity for his pri¬ 
vate network on a full-period dedicated basis, and on an on- 
demand basis for overflow situations. The customer’s total 
leased satellite transmission capacity will be dynamically 
allocated to meet his traffic demands (both as to routing and 
type—voice, data or image) based on priority assignments 
he establishes. 

Within each customer’s network, the SBS service will 
include a network management facility. This “window” into 
the customer’s communications network will enable him to 
monitor network status and performance, change traffic han¬ 
dling priorities, order changes in service and collect usage 
statistics for internal accounting purposes. 

A separate management facility will be used internally by 
SBS to monitor satellite and earth station performance, as¬ 
sist customers in designing networks to satisfy their require¬ 
ments, distribute the customer networks efficiently among 
the satellite’s transponders, and support system mainte¬ 
nance. 

The proposed SBS system will best serve organizations 
requiring large communications networks with heavy and 
dynamic traffic loads. To install earth stations at all of a 
customer’s remote sites, including those where the traffic 
would not justify an earth station, is not the intent. Rather, 
the objective is to provide an optimum system for each 
customer in which SBS ties together his traffic concentration 
points, while smaller offices—each connected by terrestrial 
services to the nearest earth station—are also brought closer 
to the center of the organization. 

SBS has strived since its inception to look beyond the 
present needs of potential customers and to address, and 
hopefully provide for, their future requirements. The PROJ¬ 
ECT PRELUDE experiments discussed previously 
touched on this future perspective by assessing new satellite 
applications in actual company operating environments. Ad¬ 
ditionally, SBS recognizes the absolute necessity for ad¬ 
vanced terminal equipment to be available in the market¬ 
place when SBS becomes operational. To assist in this, SBS 
has held a series of vendor conferences to advise about SBS 
system plans and capabilities, provide interface and service 
information, and to encourage these companies to intensify 
their development of advanced communications products. 
A conference for facsimile equipment manufacturers was 
held in December 1976 and a second conference for televi¬ 
sual equipment vendors was held in November 1977. A data 
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transmission equipment vendors’ conference is planned for 
the near future. 


ADVANCED APPLICATION POTENTIAL 

It is clear that the SBS offering must provide capability 
for traditional voice and low speed data networks, but it is 
in the area of the emerging advanced applications that SBS 
sees the greatest future potential. SBS has conducted a 
number of in-depth case studies of the future communica¬ 
tions requirements of individual potential customers and I 
would like to summarize for you briefly some of the findings. 
The advanced applications requirements we have identified 
and structured our proposed offering to serve would seem 
to be just as applicable in an international context as they 
are domestically. It should be emphasized that the leased 
services which SBS proposes to offer are for 24 hours, seven 
days per week. It is expected that this capacity, used during 
the day for conventional voice and low speed data com¬ 
munications, will allow for buffering of domestic batches of 
high speed data and facsimile for off hours shipment. And 
based on our investigations, an international requirement 
exists for this same kind of capability, taking advantage of 
off hours and time zone differences. 


High-speed data communications 

One clearly expressed requirement is for high-speed dig¬ 
ital transmission, and SBS will offer data communications 
users a fully-switched network capability among a cus¬ 
tomer’s earth station locations. Data rates include all stand¬ 
ard speeds and range up to 6.3 megabits per second. This 
means that the data base or file that now takes an hour to 
transmit at typical, high-speed 50 kilobit service will be 
transmitted in one minute. Or a one-billion-byte file (8 billion 
bits) which now has to be distributed by truck could be 
transmitted to multiple locations in half an hour. Distributed 
processing to meet operational requirements will be facili¬ 
tated by these high speeds, with the side benefits of load 
leveling among data processing facilities, redundancy of files 
for security in case of fire or accidents, and redundancy of 
processing capacity in case of machine outages. 

The high data rate networks will provide direct commu¬ 
nication links between computers, with no distance restric¬ 
tions or penalties. As a result, a large data base can be 
managed at a central computing facility, retaining the sim¬ 
plifications associated with a single control point, but with 
the capability to distribute parts, or all, of the data base to 
remote locations. This capability will add new dimensions 
to the management of data base systems and yield some 
improved management tools. These data networks will en¬ 
hance real-time central management of raw materials, pur¬ 
chases, inventories, production schedules, distribution, cost 
and pricing analyses, credit checks and postings, customer 
orders, competitor activities, etc. Many of these consider¬ 


ations are just as critical from an international perspective 
as from a domestic one. 

Facsimile communications 

SBS market investigations have shown that the availabil¬ 
ity of high-speed facsimile transmission capability can be 
expected to open up entirely new opportunities in document 
distribution from hard copy or memory as well as corre¬ 
spondence and first-class mail alternatives. To meet this 
requirement, the SBS offering will include the capability for 
digital facsimile transmission through a customer’s SBS net¬ 
work at speeds two to 20 times faster than through current 
systems. 

Teleconferencing 

A very important finding in the case studies and other 
efforts undertaken by SBS is the desire expressed by all 
levels of management for a teleconferencing capability. The 
expense, inconvenience, and time associated with frequent 
travel offer an excellent reason for holding many types of 
meetings using satellite conferencing, and these factors be¬ 
come even more significant when international travel is ad¬ 
dressed. SBS’ offering of variable bandwidth on demand 
will accelerate teleconferencing applications as a cost-effec¬ 
tive alternative to a large portion of business travel, and the 
same capacity that permits high-speed data rates and large 
voice traffic volume will be available for teleconferencing 
among multiple locations at very little incremental cost. 
Teleconferencing applications will range from slow-scan to 
full-motion color, with the customer deciding which he 
wishes to use. 

INTERNATIONAL ENVIRONMENT 

The same user needs that SBS perceives for domestic 
networks would seem to apply equally well for international 
applications. Certainly different service costs would be in¬ 
volved and the intensity of usage may not be as great as that 
for a domestic network, but the advantages to business 
operations of improved management and control through 
improved communications would be just as important. The 
benefit of being able to load level computer operations and 
take advantage of differences in prime shift hours is equally 
in evidence whether the computer centers are on the east 
and west coasts of the United States or in the United States 
and a European location. 

This audience is particularly familiar with the shift within 
the United States and other countries toward an information- 
based society, and the ability to communicate or access that 
information effectively is becoming increasingly important. 
Many things are now under way or planned which are likely 
to spur increased improvements in international data com¬ 
munications. The efforts of COMSAT, the international car¬ 
riers and foreign administrations in offering digital and 
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packet-switched services are a good example. Further, the 
plans of Intelsat and the European Space Agency to imple¬ 
ment 12/14 GHz satellite systems in the 1980’s, most likely 
with digital transmission, promise a future common carrier 
environment more conducive to the types of computer com¬ 
munications of interest to this group. 

It may be helpful to discuss for a moment the existing 
industry structure. AT&T has the monopoly for dial-up 
voice and data services and represents about 60 percent of 
this $1 billion market. The International Record Carriers 
(IRC’s)—RCA Global communications, ITT World Com¬ 
munications, Western Union International (not affiliated 
with Western Union), and TRT Telecommunications, ac¬ 
count for the remainder. The IRC’s provide dial-up and 
leased telegram/telex message services and some private line 
voice and data. Each of these carriers deals individually 
with the appropriate foreign communications agency for the 
terminating portion of the circuit, and either underseas cable 
or satellite (provided by COMSAT as a carrier’s carrier 
wholesaler) facilities are used for transmission. The Federal 
Communications Commission has designated five “gateway 
cities”—New York, Washington, New Orleans, Miami, and 
San Francisco, and all international communications must 
be routed through these locations. 

Presently, a customer with intermittent international com¬ 
munications requirements will automatically go by AT&T 
when he makes a dial-up voice or data call or will use 
Western Union for the domestic part of his record message, 
with the international segment apportioned by a pre-deter- 
mined settlements formula to one of the IRC’s. For leased 
private line voice and data circuits, the customer will deal 
directly with one of the IRC’s (or in some cases, AT&T). 
The IRC’s may “accept” the customer’s communications 
anywhere in the U.S. although they are prohibited from 
providing services in the normal sense outside the gateway 
cities. The IRC will then lease the U.S. portion of the circuit 
from one of the domestic carriers and route it through the 
nearest gateway offices to the appropriate cablehead or sat¬ 
ellite earth station for transmission overseas. In addition to 
the domestic and international circuit charges, the customer 
pays for the “backhaul” of the service from the gateway 
city to the nearest cable head or earth station, e.g., a circuit 
from Chicago to London might be routed to Washington and 
then backhauled to the Etam, West Virginia earth station 
for subsequent transmission. 

SBS has generated a great deal of interest in the user 
community, and questions are frequently asked about how 
the company plans to interconnect its services internation¬ 
ally. Because we are licensed by the FCC as a domestic 
carrier, we are restricted from providing service to foreign 
locations and must rely on the appropriate carriers serving 
those points. Additionally, SBS has chosen initially to pro¬ 
vide services to the 48 continental United States and will 
not serve Alaska, Hawaii, or Puerto Rico. Nor may we 
interconnect directly with COMSAT for service because, 
under present regulations, COMSAT is a “carrier’s carrier” 
and, as such, may deal only with licensed international car¬ 
riers. 


We will, however, keep the customer’s needs uppermost 
in our planning and work cooperatively with international 
carriers to achieve simpler and more direct connections for 
the customer’s higher speed international data communica¬ 
tions. The type of interconnect between an SBS customer’s 
domestic network services and his international locations— 
including Canada and Mexico—will naturally depend on the 
type (voice, data) and nature (speed, volume) of the com¬ 
munications and on an evaluation of costs compared with 
other alternatives. However, under present regulations, the 
customer may interconnect with the international carrier at 
his SBS earth station closest to that carrier’s gateway office, 
or he may use facilities totally outside the SBS system, just 
as he does today, if that is appropriate. 

There are some obvious limitations in this approach. Now 
let me describe briefly the type of service environment SBS 
would like to see for interconnecting its customers’ networks 
for international services. It will be over two years before 
SBS is operational and the international environment may 
be quite different then. While there is no circuit-switched 
higher speed digital service available, the need for such a 
service would seem to be clear. Many of the future appli¬ 
cations identified by SBS require periodic, bulk data move¬ 
ment, and this holds true for interconnecting with interna¬ 
tional locations. 

The FCC is presently investigating the public benefits in 
expanding the number of gateway cities in which interna¬ 
tional carriers may operate and, should a decision to expand 
result, it may prove to be best technically and economically 
for SBS to locate earth stations, conceivably its own or 
those leased to customers, in close proximity to the inter¬ 
national earth stations and interconnect directly with the 
international carriers. This would avoid the expense and 
lower speed implications associated with backhaul and 
would permit a much simpler and higher quality international 
transmission. 

The double-hop situation which will be encountered using 
a combination of domestic and international satellites may 
pose an initial problem for some users, but our conversations 
with potential customers indicate that this is far from insur¬ 
mountable. Indeed, several have either adapted, or are plan¬ 
ning, advanced data link protocols which minimize the effect 
of transmission delay and take advantage of the effective 
transmission capacity of satellites. 

Additionally, SBS would welcome the implementation by 
foreign carriers of a customer premises earth station service 
concept in their own countries, along with a wider availa¬ 
bility of high quality local distribution facilities. From our 
perspective, these are needed before uniform high-speed 
digital international networks are achieved. 

SBS would hope that foreign regulatory bodies would 
prove to be just as responsive to clearly demonstrated cus¬ 
tomer demands as has the FCC and foster an environment 
that enables a customer to assemble a uniform communi¬ 
cations network capability where there is little operational 
difference between his domestic and foreign locations. 

SBS’s efforts to date have focused on the domestic pri¬ 
vate-line communications requirements of a very specific 
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customer set—large, geographically-dispersed companies— 
and we believe that the unique service approach described 
earlier provides these organizations an innovative network 
capability for meeting forthcoming communications needs. 
It is SBS’s hope that implementation of this service within 
the United States will stimulate the international community 


with whom our customers must interconnect to move to 
provide compatible facilities and services. And SBS will 
continue to study the future requirements of the marketplace 
and make its findings known publicly. It is only through this 
open exchange of information that the potential international 
user will benefit. 





Emerging markets for satellite data 
communications in the public service* 


by JAMES G. POTTER 

Public Service Satellite Consortium 

San Diego, California 

INTRODUCTION 

The Public Service Satellite Consortium (PSSC) recently 
completed a study for NASA’s Goddard Space Flight Center 
which addresses the basic requirements of four potential 
telecommunications markets in the public service: the U.S. 
health care system, elementary and secondary education, 
American libraries, and that sector of the public service 
which is concerned with the provision of continuing edu¬ 
cation to health professionals. 1 The composite requirements 
of these and several other promising market groups are 
reviewed in this paper. 

The potential demand in 1982 for satellite communication 
capacity from public service users is projected under the 
optimistic assumption that an appropriate satellite commu¬ 
nication network will be available, a network in which nu¬ 
merous, inexpensive earth stations are located at the point 
of use. It is also assumed that there will be no “gap” in the 
development of public service satellite communications, 
which may occur if NASA’s endangered species of experi¬ 
mental satellites, such as ATS-6 and CTS, cease to function 
adequately before a growing and influential community of 
public service users graduate to operational service on com¬ 
mon carrier facilities. 

PSSC’s surveys among its members have reconfirmed 
what other studies have shown: that video applications are 
likely to predominate. 2 Seven of the ten transponders pro¬ 
jected by PSSC to support 1982 requirements are related to 
video applications. Four of these seven transponders already 
are committed to support public broadcasting. Six represent 
new markets. 

What is surprising about PSSC’s projections is evidence 
of a growing market for satellite data communications ser¬ 
vice. While projected 1982 channel capacity is modest (on 
the order of 90 MHz of bandwidth or three conventional 
satellite transponders), the potential transmission revenue 
is substantial. This potential revenue, which is estimated to 
be on the order of $100 million annually, dwarfs that which 
is likely to accrue from provision of private-line video ser¬ 
vice. 

* This work was sponsored in part by the Goddard Space Flight Center, 
Contract NAS 5-23865, under the direction of Dr. Edward A. Wolff. 


THE UTILITY OF COMMUNICATION SATELLITE 

NETWORKS 

Throughout the public service there are three recurring 
needs: improved access, maintenance of quality, and con¬ 
tainment of costs. The appropriate application of commu¬ 
nications satellite networks could ameliorate each of these 
concerns. Indeed, low cost communications is a prerequisite 
for organizational arrangements which depend upon the sub¬ 
stitution of communications for transportation to achieve 
higher productivity. 

Synchronous communications satellites have generic 
properties which make them attractive vehicles for perform¬ 
ing certain functions of interest to public service organiza¬ 
tions: 

1. Broadcasting, in which expensive audio-visual pro¬ 
grams are aggregated at a few points of origination and 
distributed to many receivers, as in a TV or radio 
network. 

2. Archiving, in which expensive computer capacity, data 
bases, audio-visual, and/or computer programs are 
concentrated at a few central locations where they may 
be accessed (and updated if appropriate) from a large 
number of remote points. An example is the computer 
utility of the Ohio College Library Center in Columbus, 
which is used by over 1100 libraries in 44 states to 
construct catalog cards and to purchase serials. 

3. Flexible routing. It is possible to allocate the available 
capacity among the possible routes in the network in 
an extremely flexible manner. Although the demand 
for private-line communications satellite circuits from 
public service organizations may be in excess of $100 
million in 1982, the geographic distribution of this 
traffic, its time distribution, the connectivity patterns, 
and the magnitude of the peak loads are unknown. The 
communications satellite will provide a cost-effective, 
“toe in the water” capable of aggregating related pri¬ 
vate-line requirements throughout a geographic region. 

THE PUBLIC SERVICE SATELLITE CONSORTIUM 
(PSSC) 

PSSC is a growing organization of over ninety non-profit 
public service agencies from the fields of education, health 
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care, library service, public broadcasting, and related inter¬ 
ests. PSSC was created in 1975 to inform the public service 
user community about the capabilities of satellite technology 
and to facilitate aggregation of public service telecommuni¬ 
cations requirements and resources. 

Most individual public service organizations cannot now 
afford access to the sophisticated information systems pres¬ 
ently (or soon to become) available. Before cost-effective 
information networks which are responsive to individual 
user requirements can become operational, it will be nec¬ 
essary to aggregate sufficient numbers of users to allow 
effective negotiations for core service requirements. 

PSSC’s role is that of a consultant and broker. It is im¬ 
portant that the established carriers have access to compre¬ 
hensive, objective information regarding the communications 
requirements of PSSC’s members and that PSSC’s members 
are aware of important capabilities which the established 
carriers can—and cannot—provide. Thus far PSSC has 
worked exclusively with NASA experimental satellites, but 
within the next year it will begin buying appropriate com¬ 
munications capacity in bulk from established carriers and 
re-selling this capacity in smaller increments to its members 
on a non-profit basis. 

PSSC has been working with the State of California and 
an ad hoc committee of federal agencies, which includes the 
White House Office of Science and Technology Policy, the 
Agency for International Development, the Appalachian 
Regional Commission, the Department of Interior, the De¬ 
partment of Health, Education and Welfare, NASA, and the 
Veterans Administration, to develop an effective program 
for accelerating the transfer of communications satellite tech¬ 
nology to the public service in the 1980s. Two alternatives 
are under discussion: (1) use of federal funds to place an 
appropriate communications payload on the Syncom IV sat¬ 
ellite of the Hughes Aircraft Company, which will be 
launched by the Space Shuttle in 1980; and (2) a competitive 
service procurement involving established carriers and per¬ 
haps aerospace companies, whereby the federal government 
will provide incentives for the private sector to meet pro¬ 
jected public service network requirements using relatively 
inexpensive satellite earth stations. There is agreement that 
the growing momentum within the public service community 
to use cost-saving communications technology must be nur¬ 
tured without interruption and that public service users must 
pay some, and eventually all, of the fair price of providing 
needed service through established common carrier mech¬ 
anisms. 


BASIS FOR DEMAND PROJECTIONS 

The projections to follow, while speculative, are based on 
two assumptions: that significant increases in telecommu¬ 
nications utilization will not occur over a five-year period 
unless: 

1. The organization obviously will benefit from changing 
its way of doing business, and 

2. The required organizational tremors will be mild. 


The public service sector has been partitioned into three 
categories: 

1. Category A—modest institutional adjustments are nec¬ 
essary and significant productivity gains are likely, re¬ 
sulting in “modest” risk to the supplier of telecom¬ 
munications services. (More specifically, the risk is 
probably still too high for a common carrier in the 
absence of federal participation to reduce his downside 
risk; but the likelihood of successful market aggrega¬ 
tion is relatively high.) 

2. Category B—the institutional requirements picture 
shows promise, but more information is needed to as¬ 
sess probable benefits and risk. 

3. Category C—major institutional adjustments are nec¬ 
essary, the risks to the communications supplier are 
high, but the possible benefits are also high. Examples 
include an electronic mail system tailored to the re¬ 
quirements of the U.S. Postal Service, or a telecom¬ 
munications/computer system addressed to core curric¬ 
ulum subjects in elementary and secondary education, 
freeing the classroom teacher to concentrate on indi¬ 
vidual problems rather than mass problems. Small 
scale experiments which were addressed to “Category 
C” problems would be appropriate for inclusion in a 
NASA experimental program, although it may be years 
before a viable commercial market develops. 

The time frame for the projections is 1982, using the best 
1976 data available to PSSC. In general, it is assumed that 
approximately one-third of the total projected telecommu¬ 
nications traffic would be carried by a communications sat¬ 
ellite if an appropriate network were available. 

The projected levels of telecommunications utilization are 
based on estimates of what it costs the organization to do 
a job now—inefficiently. PSSC assumes, for example, that 
hospitals should not have to spend 10 percent of their annual 
operating budget on record-keeping functions and is at¬ 
tempting to promote increased use of information networks 
on this basis. PSSC is assuming implicitly, however, that 
once a hospital commits to increased use of computers and 
telecommunications, it will begin using the extra capabilities 
thereby made available to perform new, unforeseen tasks. 

U.S. HEALTH-CARE SYSTEM 

PSSC concluded in its study for NASA that the three 
opportunities which are most likely to lead to extensive 
utilization of satellite communications in the U.S. health¬ 
care system involve the aggregation and sharing of: 

1. Computer power, data bases, and software for use in 
hospital information systems for accounting, billing, 
inventory control, and patient records; 

2. Clinical-support systems for radiology, cardiology, and 
pathology; and 

3. Audio-visual materials and programs for computer- 
managed instruction to support the continuing educa- 
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tion requirements of various professional groups who 
work in a hospital environment. 

The U.S. health-care system has been plagued by rampant 
inflation. Whereas the Consumer Price Index increased by 
98 percent in the period from 1950 to 1973, hospital costs 
increased by 600 percent. 3 In this same period, the federal 
contribution to health-care costs increased from 25 to 40 
percent. 4 In an effort to contain these costs, the Carter 
Administration attempted to place a ceiling on the rate of 
increase in hospital expenditures of 9 percent a year in 1977. 
Although this proposal was not approved by Congress, the 
health-care industry was placed on notice that effective, 
voluntary cost-containment procedures must be developed 
or mandatory federal regulations are likely. 

The average American hospital is estimated to spend $20 
per patient per day on record keeping, 5 an expense which 
amounted to $5.6 billion in 1976, or 10 percent of total 
expenditures. 6 A number of companies, including Techni- 
con, Shared Medical Services, Computer Sciences Corpo¬ 
ration, and McDonnell-Douglas Automation, have developed 
effective automated record keeping procedures which have 1 
resulted in productivity gains in participating hospitals. 7 Un¬ 
fortunately, present indications are that only hospitals hav¬ 
ing in excess of 200 beds can justify the expense of a dedi¬ 
cated management-information system or can afford access 
to a shared system. Yet, 70 percent of the 7,174 hospitals in 
the U.S., many of which are located in rural areas, have 
less than 200 beds. 8 The availability of an appropriate sat¬ 
ellite data communications network would permit a larger 
fraction of U.S. hospitals to access the automated admin¬ 
istrative-support systems which are now being used suc¬ 
cessfully by larger, urban hospitals. 

The advantages of an automated hospital information sys¬ 
tem include: (1) reduced clerical personnel costs; (2) reduced 
incidence of lost charges and rejected claims; and (3) im¬ 
proved cash flow. Since such a large portion of health-care 
expenditures are reimbursed by a third party (e.g., Medi¬ 
care, Blue Cross, or Blue Shield), use of central data bases 
and electronic information transfer can lead to a dramatic 
improvement in cash flow. The average payment cycle is 
reported to have been reduced from sixty days to twelve 
days in several California hospitals which recently imple¬ 
mented automated record keeping systems. 9 Other advan¬ 
tages of hospital information systems which include the pa¬ 
tient history are: (1) improved professional communications, 
which enhances continuity of care (the possibility of lost or 
incomplete records is reduced) and reduces the incidence of 
episodic care; (2) groups of patients at risk can be identified, 
and preventive medicine can be practiced systematically; 
(3) the quality of care can be audited more easily; and (4) 
the continuing education needs of the providers can be de¬ 
termined more systematically. 

The disadvantages of automated record keeping systems 
are: (1) there are potential privacy problems (although no 
system is immune from unauthorized access); (2) the relative 
strengths and weaknesses of providers can be evaluated 
more systematically, which may or may not create barriers 
to acceptance; and (3) such systems require structured input, 


which necessitates annoying changes in the working routine 
of health practitioners. 

In projecting the magnitude of the satellite data commu¬ 
nications market in 1982, PSSC makes the following as¬ 
sumptions: 

1. The rate of growth of hospital expenditures will stabi¬ 
lize at a compounded annual rate of 9 percent, as re¬ 
quested by the Carter Administration. (The rate of 
growth in 1975 and 1976 was about 14 percent, and 
total hospital expenditures amounted to $55.4 billion in 

1976. 10 ) 

2. Economies of scale resulting from use of a satellite 
data communications network will make hospital in¬ 
formation systems cost effective for hospitals having 
at least fifty beds. (Hospitals having more than fifty 
beds accounted for 97 percent of total expenditures in 

1976. 11 ) . 

3. Hospitals will continue to spend approximately 10 per¬ 
cent of their total budget on record keeping functions. 

4. The percentage of the record keeping budget spent on 
automation (given that automated procedures are used 
at all) will remain at its present value of 12 percent. 12 

5. The percentage of the automated record keeping 
budget spent on telecommunications will be 15 per¬ 
cent. 13 

6. The percentage of telecommunications service pro¬ 
vided by satellite will be 33 percent. 

The projected 1982 satellite revenue is then: 
($55.4B)(1.09) 5 (0.97)(0.12)(0.15)(0.33)=$49 million. 

Preliminary information available to PSSC suggests that 
an American hospital performs an average of sixty-two re¬ 
cord keeping transactions per patient per day. 14 Approxi¬ 
mately 270 million patient-days of service were administered 
by American hospitals having fifty beds or more in 1976. 
Assuming that the volume of traffic grows at a compounded 
rate of 9 percent, that 50 percent of the total volume is 
processed locally, that 33 percent of the remaining volume 
is carried by satellite, and that the average administrative 
form contains 8,000 bits but that only 50 percent of this 
information content needs to be transferred back to the data 
base in an average transaction, the estimated volume of 
hospital administrative traffic in 1982 is: 
(270x 10 6 )( 1,09) 5 (62)(0.5)(0.33)(8,000)(0.5)=1.7x 10 13 bits per 
year. 

PSSC has not included possible traffic associated with 
clinical support services in its estimates. Three areas which 
show near-term promise are radiology, cardiology, and pa¬ 
thology. PSSC is evaluating these areas but does not now 
have sufficient information to estimate probable demand. 

One must exercise caution when projecting demand for 
telecommunications in clinical medicine. It would be logical 
to consolidate health-care delivery throughout a geographic 
region, substituting communications for travel and treating 
patients at the lowest possible level of a hierarchical health¬ 
care system in the interest of convenience, fairness, and 
cost. Dr. Maxine Rockoff of the National Center for Health 
Services Research notes a basic fallacy in this line of think- 
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ing, however: 15 

“To the extent that the cost of care increases with each 
level in the hierarchy, avoiding referrals reduces costs, 
and benefits those who pay for care, including patients 
and insurers. But considered from the perspective of that 
‘next level up’ whose expertise is to be brought to the 
patient via telecommunications technology instead of 
having the patient referred to it, this may be no benefit at 
all. Indeed, the pecuniary interests of the ‘next level up’ 
may be best served by maximizing referrals, not minimiz¬ 
ing them.” 

A possible implication for those who wish to market tel¬ 
ecommunications services to clinicians is that one should 
concentrate on the sharing of computer and audio-visual 
resources, not scarce human resources. 

CONTINUING EDUCATION 

In a survey which was administered in the summer of 
1976, 81 percent of PSSC’s membership expressed interest 
in a network for continuing education. More recently, PSSC 
conducted a survey of the membership to determine com¬ 
posite requirements for a federally subsidized Public Service 
Communications Technology Satellite (presumably Syncom 
IV with an appropriate communications payload). In the 
period between October 17, 1977 and January 10, 1978, 
forty-two public service organizations responded in writing 
or were interviewed by senior staff. Quantitative information 
regarding the probable demand for private-line video, voice, 
and data circuits was received from seventeen organizations. 

The response was strongly oriented toward one-way 
video. If service which is comparable to that planned for 
Syncom IV (which is expected to provide an EIRP in excess 
of 43 dBW throughout the continental U.S.) is available at 
approximately $400 per transponder-hour, an estimated 
8,170 channel-hours would be utilized annually by these 
seventeen organizations. On the basis of this response, 
PSSC projects a demand for 12,000 channel-hours of one¬ 
way video in 1982, which represents an annual revenue of 
approximately $4,800,000. Although the time distribution 
and peak load of this requirement is unknown, PSSC esti¬ 
mates that three protected transponders will provide ade¬ 
quate service. 

PSSC plans to respond to this requirement in 1979 by 
using facilities of the Western Union and RCA satellite net¬ 
works. Westar will interconnect one hundred and fifty Public 
Broadcasting Service earth stations and one hundred and 
ninety-two National Public Radio stations. 16 PSSC is now 
engaged in a study for the Corporation for Public Broad¬ 
casting to develop equitable arrangements for sharing the 
public broadcasting network. PSSC plans to supplement 
coverage available from Westar using RCA’s Satcom net¬ 
work, which now interconnects approximately one hundred 
and eighty cable TV systems. 

When economical service becomes available, PSSC plans 
to develop a continuing education network which provides 


one-way television and two-way data to each classroom. 
Much of the material to be distributed will be pre-recorded, 
but an important element of the programs will be interaction 
involving use of mark-sense cards and optical card readers. 
Prior to the class, the students will receive study guides 
which contain a number of questions. The answers to these 
questions and others raised during the class will be answered 
in a multiple-choice format and relayed to the studio during 
the class. The lecturer and his assistant will receive a his¬ 
togram of responses in real-time, which will help to deter¬ 
mine what supplementary material to emphasize. Each stu¬ 
dent’s responses will be evaluated in non-real time by 
computer, and each student will be mailed an evaluation 
after the course is completed. Although this evaluation will 
be computer-generated, resources in the student’s commu¬ 
nity will be identified to follow up progress made during the 
class. 

The projected volume of data communications traffic is 
modest. Assuming that 12,000 channel-hours of program¬ 
ming are delivered annually, that there are an average of 
1,000 students per class, that the average student responds 
16 times an hour, and that 30 bits of data are transferred per 
response, only 6xl0 9 bits per year of data need be trans¬ 
ferred while maintaining individual records. The required 
system throughput is so low that the economics are signifi¬ 
cantly enhanced by relying exclusively on mark-sense cards 
and not allowing voice feedback, which tends to be ineffec¬ 
tive in large-audience situations anyway. 

A one-way video and two-way data network could com¬ 
bine the best features of pre-recorded lectures, live inter¬ 
action, and computer-managed instruction. Even with pro¬ 
duction costs as high as $20,000 per hour and transponder 
costs of $400 per hour, it would be possible to administer 
such a network at a profit while charging course fees of 
approximately six dollars per student-hour if an average of 
1,000 students per class could be attracted. 17 

EQUIPMENT MAINTENANCE 

Federal and state government agencies have extensive 
investments in equipment which is dispersed throughout the 
country. PSSC does not have adequate information regard¬ 
ing the dimensions of this requirement, but preliminary in¬ 
sight may be obtained from the experiences of the Federal 
Aviation Administration in its VORTAC (VHF Omni Range 
Tactical Communications) system. 18 In 1975 the FAA had 
a total maintenance budget of $320 million, of which 80 
percent was personnel related. Through use of telemetry 
and centralization in its VORTAC program, the FAA could 
eliminate the need for many on-site technicians and spare 
parts, which would result in a considerable savings. Studies 
by the Mitre Corporation indicate that the maintenance 
budget for the VORTAC program, which amounted to $38 
million in 1975, could be reduced to approximately $10 mil¬ 
lion annually through appropriate use of telecommunica¬ 
tions. 19 

Use of telemetry, microprocessors, and telecommunica¬ 
tions probably could effect tremendous savings in equipment 
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maintenance and load management of various resources 
(power, water, light, heat, air conditioning, oil and gas pipe¬ 
line flow, control of traffic lights and vehicular flow, etc.). 
To gain a tentative estimate of the volume of the associated 
telecommunications service, PSSC makes the following as¬ 
sumptions: 

1. The total investment in capital equipment by public 
service agencies is $100 billion, an amount which is 
growing at 5 percent annually; 

2. Five percent of this equipment is subject to savings 
through appropriate use of telemetry and centralized 
maintenance procedures; 

3. The annual cost of maintenance averages out to be 10 
percent of the capital cost of the equipment; 

4. In those cases where savings can be effected through 
use of telemetry, 15 percent of the maintenance budget 
is associated with telecommunications service; and 

5. One-third of the resulting traffic is carried by satellite. 

The resulting projected potential satellite traffic is: 
($100B)(1.05) 5 (0.05)(0.1)(0.15)(0.33)=$32 million. 

The annual bit rate is projected under the assumption that 
the average cost of this dispersed equipment is $25,000, that 
thirty-two bits of data are sufficient to address and encode 
the message, that each piece of equipment is interro¬ 
gated every ten minutes on the average, and that all of this 
information is relayed by satellite: ($100B/ 

$25K)(1.05) 5 (0.05)(32)(6)(24)(364)=4.3xI0 n Bits/Year. 

AMERICAN LIBRARIES 

Requirements to support so-called “technical services” 
in American libraries (cataloging, serials control, acquisi¬ 
tions, circulation, and inter-library loans) will be projected 
from the present budget of the Ohio College Library Center 
(OCLC), which now spends approximately $2,280,000 an¬ 
nually on leased terrestrial lines to reach 1,100 libraries in 
44 states. PSSC makes the following assumptions: 

1. Use of automated services in libraries which already 
accept the concept will grow from a present level of 3 
percent of the total operating budget to 8 percent of 
the operating budget. 20 

2. The number of libraries which use automation will 
grow at a compounded annual rate of 15 percent; and 

3. One-third of the total telecommunications traffic po¬ 
tentially could be routed by satellite. 

The potential revenue in 1982 for technical services is 
estimated to be ($2,280,000)(1.15) 5 (8.0/3.0)(0.33)=$4 million. 

5,707,828 books were cataloged in the OCLC system in 
1976, using an average of 4,800 bits per request. 21 PSSC 
projects the potential volume of satellite traffic in 1982 to be 
(5,707,828)(1.15) 5 (8.0/3.0)(0.33)(4,800)=4.8x 10 10 Bits/Year. 

To gain a qualitative appreciation of the potential demand 
for information retrieval services in 1982, PSSC makes the 
following assumptions: 


1. One percent of the U.S. population will interrogate a 
data base on the average of once a week; 

2. The average cost of each search will be $2; 

3. Fifteen percent of this revenue will be related to tele¬ 
communications; and 

4. Thirty-three percent of the traffic will be routed by 
satellite. 

The estimated revenue is then 

(220xl0 6 )($2)(52)(0.01)(0.15)(0.33)=$llM. 

Assuming that an average of ten bibliographic records are 
retrieved with each search, each of which is encoded with 
4,800 bits, the potential volume of satellite traffic is 
(220x 10 6 )(52)(0.01)(0.33)(10)(4,800) = l.Ox 10 12 Bits/Year. 

The total revenue deriving from library-related services is 
estimated to be $15 million. 


ENVIRONMENTAL MONITORING 

The National Oceanic and Atmospheric Administration 
(principally the National Weather Service), the Environ¬ 
mental Protection Agency, the U.S. Geological Survey, the 
Corps of Engineers, the Department of Agriculture, and the 
Department of Interior all have extensive requirements for 
monitoring atmospheric, edaphic, and/or oceanic data. Eco¬ 
systems International Inc., under contract to the Goddard 
Space Flight Center, determined that in 1975 these agencies 
maintained 90,714 stations in the U.S. having an average of 
four sensors apiece. 22 The annual cost of the present oper¬ 
ation is estimated to be $98.3 million and the volume of 
information is 1.36X10 11 Bits/Year. Only 6 percent of these 
platforms are remotely interrogated at the present time. 
PSSC estimates that if an appropriate satellite data com¬ 
munication network were available, environmental monitor¬ 
ing would contribute another $5 million of business annually. 
In arriving at this estimate, it is assumed that 10 percent of 
the present $100 million budget would be spent on telecom¬ 
munications services, of which half would be directed to 
satellite carriers. 


OTHER POTENTIAL MARKETS 

PSSC does not have sufficient information to evaluate a 
number of other promising candidates for service. Other 
possibilities are: 

• Department of Defense training 

• Training of other federal and state employees 

• Law enforcement (fingerprint data, automobile registra¬ 
tion and license information, LETS, and NCIC func¬ 
tions) 

• Access to data regarding eligibility for welfare benefits 

• Search and rescue/disaster relief 

• Internal Revenue Service, Social Security data transfer 

• Job bank involving the Department of Labor 
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TABLE I.—Projected Market for Public Service Satellite Communications 
(1982) 


Service Sector 

Est. 

Revenue 

Est. Volume 

American Hospitals 

$ 49M 

1.7x 10 13 Bits/Year 

Equipment Maintenance 

$ 32M 

4.3 x 10 11 Bits/Year 

American Libraries 

$ 15M 

l.Ox 10 12 Bits/Year 

Continuing Education 

$ 5M 

Three video channels 

5.8x 10 9 Bits/Year data 

Public Broadcasting 

$ 4M 

Four video channels 

Four 15 kHz audio channels 

Environmental Monitoring 

$ 5M 

6.8 x 10 10 Bits/Year 

Total: 

$110M 

Four Western Union 
transponders 

Six additional transponders 


SUMMARY OF TRAFFIC PROJECTIONS 

PSSC’s projections of “Category A” service sectors are 
summarized in Table I. Seven video transponders, which 
represent space-segment revenues of $9 million annually, 
and three additional transponders for data, which represent 
total transmission revenues of $101 million, are projected. 
Four of the projected seven video transponders will be used 
by public broadcasting, which has entered into a seven-year 
contract with Western Union. In addition, approximately 
one-third of a Western Union transponder will be used by 
National Public Radio to distribute four 15 kHz audio chan¬ 
nels to its licensees. 

In a large network of small stations, the space-segment 
revenue associated with the estimated data traffic will 
amount to approximately 20-30 percent of the total, or $20- 
$30 million. It is noteworthy that such a small amount of 
bandwidth could generate so large a revenue—given ade¬ 
quate market aggregation. 

COMMENTS ON THE NATURE OF THE TRAFFIC 

PSSC’s near-term market projections are dominated by the 
requirements of the health-care industry, both in terms of 
volume and revenue. Public education is conspicuous by its 
absence. Dr. Louis A. Bransford, Director of Service De¬ 
velopment of PSSC, concluded in the recent market study 
for NASA: 23 

“The structure of the present system of public education 
in America, both economically and programmatically, 
appears to be inconsistent with the requirements of a 
broadly-based telecommunications network. Implemen¬ 
tation of a comprehensive information network may face 
organized resistance and probably will take years to ac¬ 
complish.” 

The projected volume of satellite data communications 
traffic is on the order of 2x 10 13 bits per year. To place this 
figure in perspective, a single Satellite Business Systems 
transponder operating continuously at 50 megabits per sec¬ 


ond has a peak capacity of 1.6x 10 15 bits per year, about an 
order of magnitude higher than that required to support 
projected public service requirements, assuming that the 
peak load will be about ten times the average load. The 
earth stations which SBS allegedly plans to install have a 
capital cost of approximately $474,000. 24 The possibility of 
exchanging reduced system throughput for reduced earth- 
station cost immediately suggests itself. 

Table II was suggested by Norman Abramson of the Aloha 
Systems Project, who is an expert on “affordable” satellite 
data communications. 25 Dr. Abramson advocates use of a 
packet-switched satellite data communications network, op¬ 
erating at a speed somewhere between one and two orders 
of magnitude below the SBS network. For concreteness, 
Table II assumes a system throughput of five megabits, 
although this figure is arbitrary and not the result of a cost- 
performance evaluation. 

In a network which is composed of many point-to-point 
links which interconnect a large number of nodes, such as 
the Bell System’s Direct Distance Dialing Network for mes¬ 
sage telephone service, it is not economical to connect every 
node to every other node. Rather, the messages are routed 
through a hierarchy of nodes, using expensive switching 
machines to provide the necessary connectivity. When the 
application calls for direct connection of nodes, as in broad¬ 
casting or archiving, the additional links and switching ma¬ 
chines of a hierarchical network add unnecessary cost, com¬ 
plexity, and noise to the system. 

Dr. Abramson observes: 26 

“The natural structure of satellite communications links 
does not require the establishment of point-to-point com¬ 
munications channels in the traditional sense. A more 
natural form for satellite communications resources is a 
broadcast structure, allowing each network node to com¬ 
municate directly with every other network node. The 
communications architecture which best matches the 
broadcast structure of the satellite communications chan¬ 
nel is therefore one which starts from the premise that 
communications can proceed from many transmitters to 
many receivers in a communications environment analo¬ 
gous to that of a multi-person conference. And in the case 
of data communications it is possible to implement such 
a broadcast architecture using small earth stations.” 


TABLE II.—An Affordable Satellite Data Communications Network 


Application 

Capacity 

Capacity Using Low- 
Cost Earth Stations 

Continuous Data 

50 Mbps 

5 Mbps 

Two-Way Video 

One Conversation 

Not Applicable* 

Two-Way Audio 

800 Conversations 

80 Conversations 

3,000 Word Reports 

400 Reports/Sec. 

40 Reports/Sec. 

200 Word Messages 
Interactive Computer 

6,250 Messages/Sec. 

625 Messages/Sec. 

Terminals 

500,000 Active Users 

50,000 Active Users 


* Two-way video would not be physically impossible in this hypothetical 
network, but the earth stations probably would cost at least $85,000 apiece. 
PSSC assumes arbitrarily that earth stations must cost less than $25,000 in 
an “affordable” satellite communications network. 
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To make a satellite data communications network for 
modest-throughput requirements more affordable, Dr. 
Abramson suggests that: (1) multiple access of the tran¬ 
sponder be accomplished using Aloha-type packet-switching 
techniques; (2) the satellite transponder be designed to trans¬ 
mit at a peak power which is approximately ten times higher 
than the average power; and (3) that such transponders use 
relatively narrow bandwidths (about two MHz). With these 
modifications, the required sensitivity of two-way data earth 
stations could be reduced substantially and/or the system 
throughput could be maintained at levels which approach 
those attainable with more expensive TDMA multiple-ac¬ 
cess procedures. But if conventional satellite transponders 
are used in the early 1980s to meet public service data 
communications requirements, it is still likely that the earth 
stations can be built in lots of 50 for less than $25,000 apiece. 
Hughes Aircraft already has developed a similar earth sta¬ 
tion for operation at four and six GHz which it plans to 
market for about $25,000. 27 

CONCLUSION 

The communications capacity required to serve some impor¬ 
tant public service requirements is modest, and the potential 
revenue is significant. What is needed is a cooperative effort 
by common carriers and major public service institutions to 
aggregate the market. The federal government may or may 
not elect to accelerate the communications-technology 
transfer process by providing initial subsidies. In any case, 
PS SC will continue to focus on the requirements of its mem¬ 
bers and to impress on the carriers that there is money to 
be made by responding to these requirements. 
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Packet switching services 
for the autodin co mm unity 


by DONALD J. O’ROURKE 

United States Government 
Washington, DC 


INTRODUCTION 

The AUTODIN II Program brings the advantages of packet 
switching technology to the Defense community. The con¬ 
cepts and advantages demonstrated successfully in the AU¬ 
TODIN I store and forward message switching technology 
are not being discarded, but rather supplemented by a par¬ 


allel operating AUTODIN II in order that the best trans¬ 
mission techniques for the efficient transfer of bulk trans¬ 
action and interactive type data can be made available to 
the community of data system users. 

This paper reviews the management decision to add 
packet switching capabilities to the AUTODIN I store and 
forward message environment; examines the technology to 
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be employed, discusses the implications for future growth 
and utility in this environment, and provides a report on the 
progress toward realizing that goal. 

PLANNING FOR AUTODIN II 

Since the advent of data processing, the Federal Govern¬ 
ment has experienced a steady growth in the numbers of 
computer based information systems employing remote ter¬ 
minals. The development and implementation of the AU¬ 
TODIN I in the early 1960s was the first step in providing 
computer oriented communications network capabilities to 
this community of users. The AUTODIN I provides message 
switching store and forward communications on a world¬ 
wide basis. 

Chart I shows the presently operating switches of the 


AUTODIN I System within the Continental limits. Eight 
additional switches located on foreign soil provide the over¬ 
seas extensions of the AUTODIN I System. 

The traffic on the AUTODIN I System has steadily grown 
through the years, requiring periodic increases in the sys¬ 
tem’s capacity and the expansion in the type of services 
rendered. While the number of messages shown in Chart II 
has only grown at an average yearly rate of 5 percent, the 
type and length of messages have changed drastically over 
the past five years. The system traffic, measured in line- 
blocks (each lineblock equals 640 bits), has grown from 625 
million lineblocks in 1972 to over one billion lineblocks per 
month in 1977 (640 billion bits per month). This increase in 
traffic, plus the characteristics of the traffic itself clearly 
indicated the increased use of AUTODIN I for computer 
oriented data transfer. 

Studies by the Department of Defense and others have 
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Chart III 


shown a continuing trend toward the growth of transaction 
oriented interactive systems, in addition to the established 
requirement for remote job entry and bulk data and narrative 
transmission. A study initiated in 1972 indicated that by 1976 
there would be approximately 250 computers involved in 
on-line communications functions with approximately 8,000 
terminals associated with these systems. Projections into 
the mid-1980s indicated that within the DOD, there will be 


• HIGH TRUNK UTILIZATION 

• NO BUSY CONDITIONS (System accepts traffic-stores until transmitted.) 

• MESSAGE RESPONSIBILITY 

• SPEED, CODE AND FORMAT CONVERSION 

• MESSAGE RETRIEVAL CAPABILITY 

• MULTIPLE ADDRESSING 

• INEFFICIENT FOR MIXTURE OF INTERACTIVE AND TRANSAC¬ 
TION ORIENTED OR PRECEDENCE GOVERNMENT TRAFFIC. 

Chart IV—Characteristics of Message Switching vs. Circuit Switching 


approximately 2,500 computers with some 20,000 terminals 
involved. These studies led to an examination of the meth¬ 
odologies to be employed to meet these requirements. 
Among the earliest conclusions was the decision that to the 
fullest extent possible common-user rather than dedicated 
networks must be employed to ensure the most efficient and 
economical service possible. 


• EXCELLENT FOR INTERACTIVE AND TRANSACTION ORIENTED 
OR PRECEDENCE GOVERNMENT TRAFFIC 

• TRANSPARENT TO ALL USERS DESPITE USER PROTOCOL 

• HIGHER SPEED—LOWER ERROR RATE 

• HIGH RELIABILITY—ADAPTIVE ROUTING 

• RAPID TRANSIT THROUGH SYSTEM 

• NO MESSAGE RESPONSIBILITY 

• NO MESSAGE RETRIEVAL OR RECOVERY 

• HIGHER OVERHEAD REQUIREMENTS 

• TRAFFIC HELD AT SUBSCRIBER VIAL FLOW CONTROL 

• SMALL INTRANSIT STORAGE 

Chart IV A—Characteristics of Packet Switching vs. Message Switching 
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The economies of AUTODIN I, as a common-user mes¬ 
sage system, have been well documented. With the possible 
demise of TELPAK rates and their replacement by higher 
cost facilities, the decision by the Defense Department to 
continue their dedication to the common-user network con¬ 
cept through the implementation of AUTODIN II, is even 
more commendable in today’s world than it was even just 
five years ago when the basic design concepts of DIN II 
were originally conceived. A study of the various switching 
alternatives highlighted the characteristics of Circuit, Mes¬ 
sage and Pocket Switching. 

An examination of these summaries suggests that there is 
an advantage to maintaining a store and forward capability 
in the Federal Communications environment for message 
type traffic while adding packet switching capabilities to 
satisfy the requirements of the interactive and transaction 
oriented user. 

The trend toward interactive type traffic, as opposed to 


bulk transfer, has been clearly demonstrated (Chart III). 
While the initial traffic introduced into the AUTODIN II 
System shows that 85 percent of it is devoted to the transfer 
of bulk information, the amount of interactive and transac¬ 
tion oriented requirements will greatly expand so that by the 
early 1980s, a cross section of the AUTODIN II traffic will 
show that 45 percent will be in this category of service. 
Since packet switching, as indicated on Chart IVA, provides 
the most efficient transmission scheme for this mixture of 
traffic, the decision by the Department of Defense to imple¬ 
ment a packet switching network, is commendable. Accord¬ 
ingly, a decision was reached to acquire packet switching 
network capabilities and to have that network (AUTODIN 
II) work in conjunction with AUTODIN I. High speed in¬ 
terconnect channels will connect the two systems together. 
The message type traffic handled by AUTODIN I will be 
transferred over these dedicated links to AUTODIN II and 
transmitted over a common high speed transmission net- 
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work, thereby providing to the users the choice of two types 
of data switching facilities through a common transmission 
network. 

The request for proposal for this service was released to 
industry on November 26, 1975. An award of a contract for 
development and implementation of the system was made 
to The Western Union Telegraph Company on November 
10, 1976 with the Commercial Service Order issued January 
1977. Computer Sciences Corporation and Ford Aerospace 
and Communications Corporation are the primary sub-con- 
tractors to Western Union. (Chart V) 

The contract awarded for AUTODIN II, Phase I, covers 
the CONUS operation only. Phase II will expand AUTO¬ 
DIN II into overseas locations. Until Phase II is imple¬ 
mented, the overseas AUTODIN I System will handle data 
traffic and will be interconnected into the AUTODIN II 
System overseas at suitable gateway locations. (Chart VI) 


TECHNOLOGY EMPLOYED 

At this point it is advisable to ask “What is Packet Switch¬ 
ing?” The answer could fill volumes but, here we will re¬ 
strict ourselves to the CCITT definition (Chart VII). 

“A group of binary digits, including data and call control 
signals (address) which is switched as a composite whole. 
The data, call control signals and possibly error control 
information, are arranged in a specified format.” Essen¬ 
tially, each addressed packet occupies a transmission chan¬ 
nel for the duration of the transmission of the packet only. 
The channel is then available for use by packets being trans¬ 
ferred between other users. Thus, channels between packet 
switches are shared between many users on a demand basis, 
and it becomes possible for a user having a single link to 
engage in the exchange of data packets with a number of 
other users at the same time. This technique results in the 


740 


National Computer Conference, 1978 


iL , '• } i 'i -4 A 

vwaxirai SO 

i»toaK2s^i-SBSKPWBesfisas 


It / » c 1. .» • 

« s t£l i 


Ofcs»* i I Ijl&WiJ. { 


'JXS^UZX ^BKa SBtSSt a BUa a SSMatSS X B ^ •^ ’t^ ry^- jn^a-rfartir rxK 




P A f S/j- * 


b I 


SWITCHING IS A TRANSMISSION 


TECHNIQUE FOR SENDING DATA 

® DATA !S SEGMENTED INTOSMALL PACKETS 
(UP TO 5300 BITS EACH) 


SI SMALL PACKETS FIND THEIR WAY THRU SYSTEM 
INDIVIDUALLY AND OVER THE BEST OF MULTIPLE 
TRANSMISSION PATHS AVAILABLE BETWEEN NODES 
AT THE MOMENT C^TRANSMISSION (ADAPTIVE 
ROUTING OVER DISTRIBUTED NETWORK) 






13 ALL PACKETS ENTERING AND MOVING THRU SYSTEM 
ARE UNDER CONSTANT FLOW CONTROL - ELIMINATING 
THE NEED FOR HIGH INTRANSIT STORAGE 


Chart VII 


achievement of adaptive routing over a distributed network. 
With all packets entering the system and moving under con¬ 
stant flow control, therefore, the need for intransit storage 
is eliminated. 

To the maximum extent possible (Chart XIII), the AU¬ 
TODIN II specification and performance recognizes the 
technological advancements which evolved over many years 
from the development and operation of the Defense De¬ 
partment’s ARPA network. While the AUTODIN II System 
is based on this technology, the operational characteristics 
and the operating performance varies greatly as shown on 
the following chart. Unavailable within the ARPA technol¬ 
ogy were the requirements for multilevel classified and prec¬ 
edence type traffic, high throughput with tremendous surge 
requirements in case of national emergencies and an un¬ 
precedented 12 percent growth factor each year through the 
life of the system. 


REQUIREMENTS SPECIFICATION 

Based upon a series of studies, the DOD developed the 
following list of primary requirements: 

1. Phase I network configuration of four nodes, network 
control center and test facility, with capacity for 200 
subscribers with 13 subsystems. 

2. Modular growth capability to eight or more nodes 
with some 1500 subscribers operating in 47 subsys¬ 
tems. 

3. Node capacities: 

• 150-200 full duplex line terminations. 

• Each line can be 110 bits per second to 56 kilobits 
per second (KBPS) for subscribers and 9.6 KBPS 
to 230 KBPS for trunks. 

• Throughput form 300 KBPS to 2.5 MBPS. 
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4. Maximum packet size—5300 bits. 

5. 16-level precedence system. 

6. Non-blocking for high precedence interactive traffic. 

7. Availability of 99 percent for single-homed and 99.5 
percent for dual-homed subscribers. 

8. Minimum impact on existing subscriber equipment 
software system. 

9. End-to-end undetected error rate of less than 1 x 10~ 12 

10. Segment misdelivery rate of less than 1 x 10 -u 

11. 300 ms. max backbone delay for interactive and query 
response, based on 600 bit packets. 

12. Transparent to text. 

13. Capable of being certified to handle all levels of clas¬ 
sified traffic. 


NETWORK DESIGN 

The Network Design was predicated on the following Gov¬ 
ernment provided projected subscriber data on the present 


AUTODIN I System and the future system: 


TRAFFIC STATISTICS (BUSY HOUR) 
Average Traffic Volume, including 7.2x 10**9 Bits 


AUTODIN I 

AUTODIN I Traffic Volume Trunk 
Peak Traffic Volume Originated in 
any one second 
Computer Traffic Volume 
Terminal Traffic Volume 


4.9X 10**8 Bits 
8.lx 10**6 Bits 

5.9x 10**9 Bits 
8.5x 10**8 Bits 


Based upon these and other data, a network design of 
eight packet switching nodes (PSNs) located at the eight 
AUTODIN I sites in the Continental U.S. was developed. 
Backbone trunks between these nodes will operate at 56 
KBPS with sufficient connectivity to provide at least three 
routes out of each node. 

Chart IX shows the trunk speeds between DIN II sites 
and between AUTODIN II and AUTODIN I sites. 
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2. TRANSPARENT TERMINAL INTERFACE 

(a) Terminal Access Controller (TAC) 

(b) Interface Control Unit (ICU) 

3. TRANSPARENT HOST INTERFACE 

(a) Single Channel Control Unit (SCCU) 

(b) Multiple Channel Control Unit (MCCU) 

4. TRANSMISSION NETWORK 

5. NETWORK ENCRYPTION EQUIPMENT 

(a) Crypto Auxiliary Unit (CAU) 

(b) KG-13 

6. TEST FACILITIES AT EACH CENTER 

1. CENTRAL NETWORK CONTROL CENTER AT DC A 
HEADQUARTERS 


Chart X highlights, in an overall system configuration, 
most of the above-mentioned network elements. The dis¬ 
tributed network, using 56 KBS trunks initially intercon¬ 
nects all four of the initial system nodes directly providing 
the adapted routing principle to this packet switching net¬ 
work. A colocated DIN I facility illustrated in the southern 
switching node is directly interconnected by a high speed 
local trunk, while a remote AUTODIN I switch is terminated 
into the eastern node by a 19.2 KBS interstate transmission 
circuit. The chart also illustrates the use of the various host 
interface devices that may eliminate the need for reprogram¬ 
ming directly in the various host computers interconnect 
into the AUTODIN II System. Single-channel (1) and Multi¬ 
channel (32) Control Units are illustrated at each of the four 
packet switching nodes. The SCCU and MCCUs are located 
on customer premises directly associated with the host com¬ 
puters. 
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The system also indicates that terminal devices connected 
to the system go directly to the Terminal Access Controller 
(TAC). The TAC, however, is located within each node for 
one TAC unit can, in fact, handle 250 individual remote 
terminal devices. An Interconnect Control Unit (ICU) can 
be provided, if necessary, when terminals do not themselves 
have direct interconnectability. 

Chart XI highlights in more detail the various means to 
access the system. Host computers can provide the inter¬ 
connect protocol himself by reprogramming the host com¬ 
puter or adding a front-end device, or more simply they can 
employ either of the Western Union furnished Channel Con¬ 


trol Units (single or multichannel) and avoid any local soft¬ 
ware modification. Shown also are the various terminal 
modes terminating directly into the TAC, either directly or 
through the ICU device. Also illustrated is the ability of the 
system to terminate circuits utilizing the DDDs, the Gov¬ 
ernment AUTOVON, or the Federal Telephone System. 

Chart XII illustrates the makeup of the packet, which is 
distributed throughout the network. The packet is 5300 bits 
in length with approximately 600 bits devoted to overhead 
and 4700 bits reserved for the transmission of data. The host 
information coming through the Channel Control Units and 
terminal information coming through the Terminal Access 
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Chart XII l 


Controllers are first converted to segments in these two 
units and then converted to packets in the Switch Control 
Module just before entering the transmission system. 

NETWORK CONTROL CENTER (NCC) 

A Network Control Center monitors the operational status 
of the circuits, switching computers and user computer. It 
will assist in the diagnosing and correction of malfunctions. 
The NCC will achieve this by using the same DEC PPD 11/ 
70 Processor as is used in the PSN. This PDP 11/70 will be 
complemented with magnetic tape and disc storage, high 
speed printers and four CRT positions for operator interface. 
It will be connected to at least one PSN and possibly two 


for redundancy connection will take place via communica¬ 
tion links. 


PATCH AND TEST FACILITY SUBSYSTEM 

The second segment of the PSN is the Patch and Test 
Facility. At the AUTODIN I colocated sites, the AUTO¬ 
DIN II PTF will be integrated with the existing AUTODIN 
I PTF. In these combined facilities monitoring function will 
be performed by the SCM. All monitor points that have a 
direct bearing on circuit reliability and quality will be auto¬ 
matically scanned, and the scan results sent to the SCM. All 
control commands required to restore or maintain opera- 
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Chart XIII 


tional circuit integrity will be formatted and transmitted by 
the SCM (either automatically or upon request from the tech 
controller) and will be executed by the PTF equipment. 
(Chart XIII) 


CONCLUSIONS 

Packet switching technology as added to the AUTODIN 
provides an example of the adaptation of new technology to 
an established environment. The user of this and the earlier 
implemented message switch technologies now has an op¬ 
portunity to employ that unique capability that best satisfied 
his data communications need. 

With the interconnect capability to be established between 


the AUTODIN I and AUTODIN II, the user can select that 
optimal network which meets his immediate need. 
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INTRODUCTION 

A national computer network linking educational and re¬ 
search institutions is frequently proposed as a means of 
making available specialized computing resources, minimiz¬ 
ing duplicate software development, and making more effi¬ 
cient use of the nation’s total computing resources. How¬ 
ever, a number of economic, political, and organizational, 
rather than technical, issues must be resolved before such 
a network is constructed. A Network Simulation Project, 
supported by the National Science Foundation, has been 
investigating these issues. Now in its third year, the project 
has gathered and analyzed a variety of data contributed by 
representative institutions concerning their computing re¬ 
sources, user profiles, and computing policies and practices. 

Results of the modeling and data analysis support the 
arguments for operational economy, software development 
and maintenance economies, specialized service availability, 
and other proposed benefits of resource sharing at a national 
level. On the other hand, a number of barriers have been 
identified that must be addressed. These include institutional 
constraints on balance of payments (both positive and neg¬ 
ative), institutional inabilities to expand service offerings to 
the network, difficulty of account control, and several net¬ 
work organizational and management problems. 

The results of simulation and gaming exercises involving 
institutional administrators and other personnel were ob¬ 
tained too late for inclusion in the proceedings. These will 
be discussed at the presentation. 

THE PROBLEM 

Decision makers at educational and research institutions 
are grappling with increasingly difficult questions concerning 


* This work is sponsored by the National Science Foundation under Grant 
Number MCS75-03634, “A Simulation and Gaming Project for Inter-Institu¬ 
tional Computer Networking” (Network Simulation Project). 


the best means to satisfy the computing requirements of 
students, faculty, and staff members. The computing needs 
of these groups are increasing in variety and complexity, as 
well as in magnitude; and the means of satisfying these 
needs are becoming broader. Consequently, it is clear that 
attempting to meet all computing requirements “in-house” 
is no longer feasible. Technological advances in computing 
hardware will not, in themselves, solve the problem, for soft¬ 
ware and data base development, maintenance, and user 
support represent an increasingly larger part of the total cost 
of computing. As the variety of computing requirements 
increases, the non-hardware portion of computing costs can 
only escalate. 

One mechanism for alleviating some of the problems 
posed is the computer network. Not only can networks 
provide ready access to a wide variety of available and 
supported software, but they can make these resources 
available very cost effective. 1 In 1972-73 EDUCOM,* with 
the financial support of the National Science Foundation, 
organized a series of General Working Seminars 2 on the 
subject of computer networking in higher education and 
research. Participants included university administrators, 
computer center directors, users from key disciplines, and 
computer scientists. Both general and specialized working 
sessions were organized to investigate many aspects of net¬ 
working. 

In session after session, regardless of their professional 
roles, participants came to similar general conclusions— 
namely, that it is now technically feasible to create a net¬ 
work linking computer facilities at colleges, universities, and 
research institutions around the country. Although many 
technological problems remain, the primary difficulties con¬ 
fronting such a network were felt to be non-technical in 
nature, involving economic, political, and organizational 


* EDUCOM is a consortium of more than 250 colleges, universities, and non¬ 
profit organizations that serve higher education. It was founded to help its 
members make the most effective use of computer and communications 
technology. 
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considerations. Obtaining a clear understanding of these fac¬ 
tors before undertaking any large-scale networking devel¬ 
opment was seen as critical. It was this judgment that led 
to the initiation of the current research project. 

THE RESEARCH PROJECT 

While existing networks provide an indication of the costs 
and technical characteristics that might be associated with 
a national network, they cannot indicate how member insti¬ 
tutions would be affected by economic, organizational, po¬ 
litical, and intellectual considerations; how users would be 
affected by the availability of multiple resources at a variety 
of prices; how “balance of payments” problems (both def¬ 
icits and surpluses) would impact institutions; and how a 
network might be impacted by institutional actions. 

No existing network has the necessary characteristics to 
permit the study of these issues. Although the ARPA net¬ 
work, for example, offers interconnection between comput¬ 
ing facilities at many universities and research institutions 
throughout the country, the single-source nature of the fund¬ 
ing for most of the computing activities removes the source 
of many of the problems that need to be investigated. 

The use of an actual network to examine many of these 
issues poses a number of other problems. Experimentation 
on a national basis would be extremely costly, would take 
several years to conduct, would be severely restrictive in 
the approaches and alternatives that could be investigated, 
would be disruptive to the normal operations of both the 
network and its institutions, and would require an inordinate 
commitment of energy and personal time from key individ¬ 
uals across the country. 

In recognition of these difficulties, a simulation apporach 
was selected to investigate some of the behavioral and policy 
issues of such a computing network. Focus is on the poten¬ 
tial impact of the network upon member institutions, as well 
as the impacts of institutional decisions and policies upon 
the network. The simulation approach provides flexibility as 
to the types of networking situations that can be examined, 
and allows for the testing, at a relatively low cost, of a 
variety of alternatives. For example, the model can be used 
as an analysis tool to investigate the impacts of a wide 
variety of behavioral assumptions. It can also support a 
game, played by institutional representatives, to investigate 
questions pertaining to likely behavior encountered in a net¬ 
working environment. 

Initially, a six month planning study was conducted by 
eight individuals knowledgeable in the areas of model build¬ 
ing, gaming, economics, resource administration, and edu¬ 
cational and research computing. 3 The present Network 
Simulation Project is a three-year project implementing that 
plan. 

The first phase of the project was devoted to basic data 
collection and analysis. Each of eighteen participating insti¬ 
tutions contributed data on their computing resources (hard¬ 
ware and software), pricing schedules, priorities, turnaround 
levels, demand, and user behavior profiles. The network 
simulation model was developed and used with the accu¬ 


mulated data base to conduct a series of preliminary inves¬ 
tigations. Details of the phase one model development and 
findings have been reported in References 4 through 10. 

The second phase of the project was devoted to including 
computing policy information in the institutional data base, 
and to refining the simulation model. Visiting teams inter¬ 
viewed administrators, users, and computer center person¬ 
nel at each participating institution. The computing policies 
and practices in effect at each institution were deduced from 
this information and subsequently confirmed with the par¬ 
ticipants. This updated data base is currently being used in 
a series of further experiments using the network simulation 
model. The data has also been used to support many of the 
analyses discussed below. 

The third phase of the project, currently in progress, has 
converted the simulation model into a gaming version with 
a conversational interface. Using this version, both a remote 
and a centralized set of gaming experiments are being con¬ 
ducted. Personnel from each of the eighteen institutions are 
actively participating, each representing the “best” interests 
of their home organization. In the remote experiments, each 
institution is operating with a “straw network,” a static 
network using programmed decision rules for all institutions 
(except for the player’s own institution). This provides for 
controlled experiments in which responses to particular 
types of circumstances and environmental conditions can be 
tested. The centralized experiments will then be conducted 
during an intensive live three-day session, which will permit 
the investigation of dynamic inter-institutional interactions.* 


BENEFITS OF A NATIONAL NETWORK 

Analyses of the resources and policy data collected from 
the participating institutions, as well as investigations using 
the network simulation model, have revealed a variety of 
positive factors associated with a national network. Several 
of these are summarized here. 

Operational economies 

The availability of network computing resources can lead 
to the combining of facilities, a reduction in size or range of 
offerings at some computer centers, and better utilization of 
the remaining facilities. Any of these alternatives can result 
in operational economies. For example, Harvard has been 
able to significantly reduce the size of its central computer 
facility, relying heavily on remote services obtained via tel¬ 
ecommunications access. Similarly, Harvard’s primary sup¬ 
plier, MIT, was able to justify an expanded and better staffed 
operation because of the guaranteed outside usage (income) 
from Harvard. This results in a more powerful and cost 
effective service to the MIT community. Obviously, a na¬ 
tional network will not enable everyone to participate in this 


* These experiments will have been completed by the NCC, and the results 
will be discussed during the authors’ presentation at the Conference. 
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manner. However, the opportunity is there for institutions 
in appropriate circumstances. 


Software and data base economies of scale 

For a variety of services, software and data base main¬ 
tenance costs—including associated user services such as 
documentation and support—do not vary significantly with 
usage. Since a centralized center permits spreading these 
costs over a larger number of users, it can provide substan¬ 
tial economies. As hardware costs continue to decline, and 
software-related costs continue to increase, these economies 
will become increasingly important. 


Availability of specialized services 

One of the great advantages of a network is that it allows 
specialization while still exploiting economies of scale. A 
computer center that offers a specialized service can attract 
enough network users to justify the high fixed costs of main¬ 
taining the service. Thus, the availability of such services is 
likely to increase rapidly. Networks can, in this way, com¬ 
plement localized centers. A computer center can choose to 
provide relatively standard services for its own users, plus 
a few specialized services for the national market, and then 
look to the network to obtain specialized services that it 
does not provide. 

Data collected by the project indicate that there is a vast 
array of specialized services already available that are ap¬ 
propriate for a network environment. However, it is likely 
that these services only represent a small fraction of the 
total that would be developed once the potential usage base 
was realized. 


Economies of comparative hardware advantage 

Not every computer system can process every type of job 
with the same relative efficiency. Hardware and operating 
system differences, as well as local modifications to com¬ 
puter systems, have resulted in a wide variety of configu¬ 
rations, each having different processing characteristics and 
capabilities. As a consequence, systems have different rel¬ 
ative advantages. If one were to take the processing that is 
being performed at a set of institutions, and redistribute it 
among these same institutions on the basis of the compar¬ 
ative advantage of their computer systems, net computer 
resource usage would decrease relative to capacity. 

The EDUCOM Planning Council* 11 has developed a se¬ 
ries of benchmark programs that illustrate the point dra- 

* The Planning Council is a cooperative effort of twenty-two universities that 
have decided jointly to “determine how the computing needs of their insti¬ 
tutions can best be met, assess the means of achieving efficient and effective 
resource sharing, and develop a national ‘facilitating network’ linking com¬ 
puters at colleges and universities throughout the United States.” 


matically. 12 Although subject to all the usual problems of 
“representativeness,” the benchmarks have been run at 
each of the participating institutions. They provide a system- 
by-system comparison of the difficulty of converting a pro¬ 
gram from one site to another, the computer resources re¬ 
quired to execute the programs, and the cost to execute the 
programs. 

In the tests, there were several examples of price differ¬ 
ences on the order of 10:1 and even 20:1 for the processing 
of a given job. Although the lowest priced facility is not 
always the most effective when all factors (e.g., turnaround 
and user support) are considered, these price differences are 
attractive enough to encourage individual users to take 
advantage of the comparative advantages of different sys¬ 
tems, thus resulting in major economies. Admittedly, prices 
are not always directly tied to costs, 13 and many of the price 
differentials would diminish in a free market environment. 
However, the institutions involved believed that their pric¬ 
ing was an accurate reflection of true costs at their facility 
and that substantive changes would be unlikely. 

The identities of the “lowest cost” and “highest cost” 
institutions were not constant across the benchmark jobs. 
That is, one institution was more efficient for some job 
classes but not for others. Thus, the trading opportunities 
are not one-sided and there should be a reasonable distri¬ 
bution of usage. This situation is analogous to the economies 
of comparative advantage in international trade. 14 

National software marketplace 

At the present time there is no simple marketing or dis¬ 
tribution mechanism for available software. To market a 
product, the developer must prepare user documentation 
and installation instructions, must prepare versions for dif¬ 
ferent computer systems, must be prepared to consult on 
implementation problems, and must develop and distribute 
maintenance updates for all systems. For the person or 
organization that is not in the business of software devel¬ 
opment, this is a significant barrier. Thus, most local de¬ 
velopments are not exported, forcing duplicate development 
if the capability is to be obtained at all. 

The existence of a national network would change much 
of this, since prospective users would be able to use the 
software on the developer’s own system. All users would 
run with the same program on the same system, permitting 
a single set of documentation materials and a single main¬ 
tenance operation. Thus, the burdens upon the developer 
(who is often more interested in his discipline or research 
subject than he is in software development and marketing) 
are greatly reduced. 

A national market would also make it possible to apply 
usage royalties. Not only could this be used to provide an 
economic incentive to develop and support good software, 
but it could also provide a mechanism to support the addi¬ 
tional costs of polished documentation and user support. 
Thus, the network framework could support greater sharing 
of software development and a faster spread of beneficial 
software. 
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Management improvements 

Although computing has grown into a substantial business 
at many institutions, the effectiveness of the management 
and operation of the local computer center has not always 
developed analogously. In all too many instances, our data 
and on-site interviews revealed that large computational fa¬ 
cilities are being run as if they were small service facilities 
(which indeed they were not very many years ago). Yet 
large computing operations are a business and must be rec¬ 
ognized and operated as such. 

The existence of a national network would bring a meas¬ 
ure of comparison and competition to all participating insti¬ 
tutions. Not only would there be price comparisons by pro¬ 
spective users, but there would also be service, support, 
and reliability comparisons as well. Poorly managed facili¬ 
ties would find themselves in a poor competitive position. 
If local administrations permit free choice, such facilities 
would be forced to either upgrade operations or lose busi¬ 
ness to the better-run facilities. From the viewpoint of many 
individual facilities, this would be a traumatic experience. 
In any event, though, the user will perceive more effective 
service. 

Positive attitude 

A positive attitude is critical to the success of any en¬ 
deavor. Surveys of regional networks in higher education 15,16 
showed that many networks succeeded (or failed) in large 
part becuase of the beliefs of key participants that the net¬ 
work would succeed (or fail). In the case of a possible 
national network, our interviews with administrators, com¬ 
puter center directors, and others revealed a consensus of 
positive attitudes. Administrators envisioned cost savings, 
increased research potential, and better overall service to 
their community. Computer center personnel anticipated a 
wider audience for their offerings, greater service to internal 
users, and elimination of some costly local services. Users 
felt that “competition” would eventually bring more cost 
effective service to them, and that the expanded availability 
of suppliers and offerings would be beneficial. Although 
such attitudes do not, of course, guarantee the success of a 
networking activity, they are a vital ingredient. 

Preliminary model results 

A number of explorations have already been conducted 
using the basic simulation model. 4,9 For these tests, per¬ 
ceived decision and behavior patterns of all sites were pre¬ 
programmed (as compared to the gaming studies in which 
real people will be making decisions at each site). Although 
results are not yet conclusive or comprehensive, indications 
are positive in all areas tested. The network appeared stable 
in the face of large shocks; there is the opportunity for large 
cost savings to users; communications costs are relatively 
low for many of the likely early usage categories; the more 
“attractive” supplier sites seem to have sufficient available 


capacity; and the different hardware configurations appear 
to present useful differences in performance features. More 
sophisticated and revealing research experiments, as de¬ 
scribed in a later section, are now being conducted. 


BARRIERS TO NETWORKING 

Our analyses to date have also revealed a number of 
barriers to networking. Although adverse, these barriers are 
not insurmountable. However, they must be addressed by 
any network undertaking. 

Institutions are unprepared to modify computer facilities 

Many institutions are viewing the network as a mechanism 
for selling “a little” excess capacity and for obtaining “a 
little” extra revenue. They are not prepared to serve the 
growing computing requirements of a national user com¬ 
munity. That is, most institutions, even if they had a highly 
sought after set of services, would be very unlikely to install 
additional large-scale computing equipment solely to serve 
the needs of network users. In many cases, the charter of 
the institution or the laws or policies under which it operates 
preclude such an expansion. 

On the other hand, most institutions believe (in most 
cases, quite strongly) that they must have a major central 
facility on site if they are to maintain their research and 
educational position. Further, few institutions have a great 
deal of short-term flexibility to reduce their scale of com¬ 
puter operations, even if they deemed it advisable. There 
are purchased equipment, long-term leases, personnel com¬ 
mitments, state restrictions, and other considerations that 
make rapid change difficult. 

Thus, we have a situation in which facilities are unlikely 
to be expanded or contracted significantly. This implies that 
the imports and the exports of computer services at a site 
must either be fairly closely matched, or must be small in 
magnitude relative to the total computing budget. We have 
made several simulation runs using a policy of “no imbal¬ 
ance of payments” on the part of each institution. The 
results do not bode well for network vitality, for the work 
flows that can develop under these assumptions are rather 
limited. 

Lack of incentives for sharing 

One of the serious problems that is likely to constrain the 
growth of network computing is the lack of strong incentives 
to encourage sharing. Incentives are missing for both the 
buyer and the seller of services, and at the level of both the 
individual and the institution. 

On the buying side, individual users are likely to be frus¬ 
trated by shortcomings in documentation, consulting, and 
other support services; by the unreliability of existing soft¬ 
ware and data bases; and by difficulties in locating and 
accessing appropriate remote resources. At the institutional 
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level, money spent on remote services by internal constit¬ 
uents is usually perceived as “real” money resulting in a 
financial drain to the institution; whereas the cost of local 
services is largely fixed and does not impose significant 
incremental costs in the short run if usage is increased (nor 
incremental savings if it is reduced). Consequently, there is 
often a reluctance to permit access to outside services. 

Unfortunately, the situation is just as difficult on the sup¬ 
plying side. Faculty members and other individuals who 
create programs and data bases typically receive neither 
financial benefit, nor professional rewards such as those 
associated with the publication of a paper in a prestige jour¬ 
nal. Consequently, there is little incentive to undertake the 
incremental work required to document, remove errors, and 
otherwise convert a successful “local” resource into one 
that is acceptable for a network environment. 

Similarly, institutions lack strong incentives as suppliers. 
Economies of scale in computer hardware have traditionally 
been one of the primary justifications for concentrating a 
variety of computing functions into a single large central 
processor. However, except for a few quite specialized 
cases, hardware economies are no longer an important mo¬ 
tivation. Minicomputers now compete economically with 
the large machines for raw cycles, and any significant econ¬ 
omies of scale for very large jobs can usually be obtained 
regionally, if not locally. Further, the cost of the central 
processor as a fraction of total computing costs is shrinking 
rapidly. However, economies of scale still hold for large, 
specialized processors in such fields as physics and meteo¬ 
rology. Thus, for example, the ILLIAC IV computer at 
NASA Ames is likely to become a major network resource. 
Most institutions, therefore, have little to gain from the use 
of network demand to justify a larger central processor. 

Unfair competition 

Not all institutions that might belong to the network are 
bound to operate under the same pricing regulations. For 
example, most educational and non-profit research institu¬ 
tions performing government-funded research are bound to 
operate all services on a fully cost-recovered basis. On the 
other hand, government laboratories often acquire comput¬ 
ing equipment with capital funds which do not appear in 
their cost base. Thus, they can and do offer services to other 
users at marginal or operational cost. Further, commercial 
organizations could offer services via the same or a com¬ 
patible network. Loss leaders and other types of promo¬ 
tional prices could be established. Thus, institutions com¬ 
mitted to a large central facility could find their user base 
severely eroded by price competition, their ability to com¬ 
pete on price severely restricted, and their ability to reduce 
expenses limited. 

Guarantees 

Institutions that might potentially offer computer services 
over a national network have shown little willingness to 


provide service guarantees to outside users, except in some 
very special cases. Their basic operating policy is “our users 
first” (and properly so according to their charters). Thus, if 
demand were to exceed available capacity, the local user 
would be served and the network user would often be denied 
service. This means that other institutions and users cannot 
depend upon the continued availability of service. Although 
it is unlikely that abrupt large-scale cut-offs would ever 
occur, the possibility of disruption is a significant concern. 

Looking at the problem the other way, most users are not 
in a position to make long-term usage commitments. How¬ 
ever, without such commitments, most supplier installations 
are unwilling to assume the risk inherent in expanding ca¬ 
pacity or services. 

Usage barriers 

While price (economics) is a powerful motivator with re¬ 
spect to users shopping for and switching to outside ser¬ 
vices, user behavior profiles developed at each institution 
confirm that current computing technology limitations are 
significant. First, there are differences in command lan¬ 
guages, file procedures, accounting methods, and the like 
from site to site. Even such things as “standard” FOR¬ 
TRAN compilers have differences and unique “quirks.” 
Learning to work with many different sites is such a personal 
burden for most users that only a small number of them are 
even potential network customers. 

Second, software conversion is time-consuming and 
“unrewarding.” Once a site is selected as the supplier, the 
software will likely be developed and then executed at that 
site for the duration of its useful life. Evidence indicates that 
even sharp differences in price or service levels are unlikely 
to cause any appreciable shifting of existing programs be¬ 
tween sites. 

A third barrier is information. The user needs to find out 
“what is offered where,” how the system operates, what 
the pricing structure is, and so forth. Further, he needs a 
mechanism for determining how effective each site is for his 
own computing needs. Unless this fairly costly process can 
be written off over a significant amount of future computing, 
it is not cost-effective to consider many other supplier sites. 
To the extent that there is no central organization to collect, 
keep current, and distribute such information, many users 
will be unable to find alternate computing sites that would 
be beneficial. 


Network overhead 

Participation in a network is not without cost. In addition 
to the base cost of communications between sites, interfaces 
to the network communications facilities have to be imple¬ 
mented for each host computer; billing and accounting rou¬ 
tines have to be modified so that this information can be 
provided to each user, institution, or network clearing house 
as appropriate; administrative structures must be imple¬ 
mented; and standards and interface procedures must be 
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provided and enforced. For an institution that only wants to 
distribute or gain access to a small volume of services, these 
requirements could be formidable. 

Security 

Current systems provide inadequate protection against 
malicious or unintentional access to system resources. Until 
this situation is corrected, most institutions will choose not 
to maintain sensitive data on remote storage devices acces¬ 
sible via a network. Applications involving personnel or 
student records, detailed financial data, or other confidential 
information will thus be unlikely candidates for early net¬ 
work service. 

Another aspect of security is protection against unauthor¬ 
ized use of network resources. When an institution provides 
its own computing services, its total financial risk is limited 
to the expenses of running the computer center. In the case 
of a network, however, an institution’s potential consump¬ 
tion of resources could be virtually unlimited. 

Government, regulatory, and tax issues 

There are a number of unresolved legal and regulatory 
issues that must be faced. For example, can a nonprofit 
educational institution freely sell computing services and 
still retain its nonprofit status? How will the federal govern¬ 
ment requirement that it receive the most favorable price 
offered to any buyer impact the ability of institutions to use 
flexible prices in rationing their resources more efficiently? 
Will the constraint imposed by many state legislatures that 
both the sale and purchase of computing resources at state 
universities be limited to in-state facilities exclude these 
institutions from national network participation? 

Many institutions, because of educational discounts as¬ 
sociated with equipment acquisitions, institutional charters, 
or tax regulations, are unable to process particular types of 
work. Commerical work, for example, is prohibited or se¬ 
verely limited at many sites. Although an institution can 
exercise control to some degree over its own local users 
(i.e., forbid commercial accounts), it is very difficult to 
exercise such control over unknown network users. Thus, 
a commercial user that can’t use the low-cost service at 
institution “X”, might be able to obtain system entry at less 
restrictive institution “Y” and hence be able to access in¬ 
stitution “X” as a “Y-user.” This is a serious concern to 
some institutions. 


Unfulfilled expectations 

While many administrators, computer center directors, 
and others have very favorable attitudes toward networking 
(as mentioned earlier), there will be many disappointments 
due to inappropriate expectations. For example, our data 
show that most institutions believe that they offer attractive, 
economical services and that, in a free market environment, 


they would be net sellers to the network. It is not the po¬ 
sition of this research to critique the individual viewpoints; 
however, it is obvious that not all institutions can be net 
sellers. On numerous occasions in the regional networking 
programs undertaken in the late 1960’s, 15 such disappoint¬ 
ments resulted in changes in attitudes and the erection of 
artificial barriers to trade that adversely affected the net¬ 
working endeavor. 


LIKELY CHARACTERISTICS OF A NATIONAL 

NETWORK 

Although our research has not been completed, a number 
of indications have been found as to what the likely char¬ 
acteristics of a national network might be. Some of the areas 
being investigated more thoroughly are listed below. 

Network development 

1. Usage Growth—Network usage is likely to increase at 
a steady rate without any sudden surges in growth. A rela¬ 
tively long transition to a network environment is expected 
for buyers to gain experience with network services, for 
institutions to adjust their internal capacities in conformity 
with changes in demand patterns, and for suppliers to de¬ 
velop more powerful aids for assisting remote users. 

2. Evolution—The network is likely to evolve gradually, 
rather than be implemented in one major step. To adapt to 
changing needs and technology, the design at each stage of 
implementation should be kept as open-ended as possible 
and should avoid large fixed costs (e.g., costs should be 
incurred in proportion to the level of network activity). 

3. Volume of Usage—Among participants there will be a 
wide variance in network activity, with individual sites rang¬ 
ing from no use to total dependence. The most optimistic 
present estimates suggest that network dollar volume will 
represent less than five percent of the total computing done 
by member institutions. Note, however, that even at very 
low per unit levels, aggregate flows from all members could 
ultimately be very large. 

Network characteristics 

1. Hierarchy of Networks—It is likely that there will be 
a hierarchy of networks available to users, each serving a 
different need. These would include local mini-computer 
networks, campus centers, regional and state networks, na¬ 
tional networks, and commercial facilities. 

2. Autonomy of Members—Colleges and universities in 
this country are largely autonomous and show no intention 
of weakening this autonomy in their quest for computing 
efficiency or power. This strongly suggests that a successful 
national network for higher education must be largely de¬ 
centralized. Each institution should be allowed to choose 
which services it will buy, which services it will sell, and to 
whom it will sell them. An institution should also be free to 
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set both the prices and the quality of services it provides to 
others. 

3. Central Facilitating Function—Although decision mak¬ 
ing is likely to be largely decentralized, certain facilitating 
functions need to be provided centrally. Included in such 
functions are billing mechanisms, information on services 
and prices, standards, security procedures, and provision of 
network-wide user services. 

4. Financially Self-Supporting—A national network of au¬ 
tonomous members must be self-supporting and will likely 
rely on a free market mechanism to provide incentives for 
both suppliers and buyers. Although financial aid from gov¬ 
ernment agencies and private foundations might be available 
to accelerate the development process, the operational net¬ 
work must be economically viable on its own without sub¬ 
sidy. 

5. Evolution of Successful Suppliers—The data collected 
to date indicate that many of the most popular sites will 
“spin-off’ their computing facilities (or at least that portion 
serving external customers) to a separate, not-for-profit or 
even for-profit subsidiary. Under such a scenario, many 
universities and research institutes would reduce their own 
local computing capabilities and either use or provide ser¬ 
vice from a “computer service business.” Consequently, 
what started as a network linking the computer facilities of 
a number of research and educational institutions, might 
well turn into a network linking a number of university- 
owned service bureaus. Several examples of such “spin¬ 
off’ activities already exist. The best known are the Dart¬ 
mouth Time Sharing System (DTSS), the Ohio College Li¬ 
brary Center (OCLC), and the Stanford Library System 
(Ballots). 

Network usage modes 

1. Functionally Distributed—Economics favors local or 
regional computing for most services, with the national net¬ 
work providing specialized services that cannot be econom¬ 
ically supplied or maintained locally. Most suppliers will 
therefore concentrate on providing a relatively few special¬ 
ized services that exploit their comparative advantage. 

2. Specialized Services—The specific specialized services 
that are likely to exist in a network environment are difficult 
to predict in advance, although communications constraints 
are currently forcing an emphasis on services with low to 
medium volumes of data transfer. Candidate specialized ser¬ 
vices will be those that require some combination of com¬ 
plex software, large data bases, sophisticated support, or 
specialized facilities. Examples include library services, bib¬ 
liographic services, person-to-person communications 
(“electronic mail”), chemistry and census data bases, econ¬ 
ometric forecasting systems, and high energy physics com¬ 
putations. 

User support 

One of the barriers to network usage mentioned earlier 
concerned the difficulty of appropriately supporting remote 


users. In addition to all of the forms of user services cur¬ 
rently provided, the network will likely also offer some 
combination of the following: 

• Both local and central consultants who are knowledge¬ 
able about services available over the network, or at 
least can facilitate the acquisition of information. 

• “Circuit riders” who periodically visit remote sites. 

• Remote consulting services provided on-line over a 
“hot-line” or by means of an “electronic mailbox.” 

• High quality documentation designed to serve the in¬ 
dependent remote user. 

• Computer-assisted training aids. 

• On-line information data bases to assist users in locating 
remote services appropriate to their needs. 

CONFIRMATION EXPERIMENTS 

The findings discussed above are based upon the resource 
and policy data that have been collected in the first two 
phases of the Network Simulation Project and upon the 
initial simulation experiments. In order to confirm the valid¬ 
ity of these findings, a further series of experiments is cur¬ 
rently under way. 

The first set of experiments is being conducted by project 
staff using the basic network simulation model together with 
site-specific data bases and behavioral information. General 
topics such as the following are being investigated: 10 

• Expected network performance 

• Site specialization 

• Network stability 

• Network resource-sharing potential 

• Communications costs 

• Service pricing policies 

• Provision of special services 

• Network equilibrium conditions 

• Quality of network information provided 

• Network growth effects 

These investigations will continue in parallel with the par¬ 
ticipant experiments (see below). 

The second series of tests (referred to as the “straw 
games”) involves active participants from each institution 
playing against a programmed network (“programmed” is 
used in the sense that behavior patterns, decision rules, 
offerings, and so forth are pre-entered for all sites except 
one). This permits a series of experiments to be performed 
testing institutional reaction to controlled environmental cir¬ 
cumstances. Typical experiments in this series involve: 

• Reactions of supplier sites to increasing levels of de¬ 
mand from the network 

• Latent demand for various proposed science informa¬ 
tion services 

• Effects on supplier sites of large erratic demands from 
the network 

• Shocks, such as major suppliers leaving the network 
unexpectedly 
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• Commercial vendors joining the network 

• Reactions of supplier sites when they lose usage 

• Effects of small schools attempting to satisfy much of 
their computing needs via the network. 

• Implications of sites reducing their in-house facilities as 
network dependence increases. 

The third series of experiments (the “live game”) will be 
conducted at a central site with live players from each in¬ 
stitution on the network playing simultaneously. This will 
permit a series of experiments to test inter- and intra-insti- 
tutional interactions on a dynamic basis. 

Typical experiments in this series involve: 

• Institutional budget crises 

• Unexpected federal edicts that affect the way many 
institutions have been using the network 

• Changes in tax regulations for non-profit institutions 

• Availability of a variety of competitive science infor¬ 
mation services 

• Network “shocks” 

• Technological breakthroughs in communications cost, 
capacity, interfaces, etc. 

• Breaches of security 

• Impact of changing network administrative structures 


SUMMARY AND CONCLUSIONS 

The studies to date support the expectation that establish¬ 
ment of a national resource-sharing computing network for 
higher education and science research is a formidable task. 
The barriers described in this paper will not be easily over¬ 
come. However, all evidence points to a potential high pay¬ 
off from, if not a vital requirement for, such a network. In 
particular, the specialized hardware and software resources 
that would be accessible could not be made broadly avail¬ 
able via any other mechanism. 

Early indicators are extremely favorable. Both potential 
suppliers and potential users see it in their self-interest to 
participate in networking activities. Attitudes toward net¬ 
working are positive and constructive, and administrators at 
the highest level are becoming actively involved. The likely 
deciding factor is whether or not short term pay-offs can be 
made attractive enough to sustain interest during the difficult 
development period. 

Much research still remains to be done, and the efforts 
described in this paper are continuing. The various gaming 
studies are enabling decision makers at participating institu¬ 
tions to gain experience with operational considerations in 


a network environment. As their experience matures, they 
are becoming more confident in their ability to properly 
represent their institutional needs in such an environment. 
The above experiences are also permitting more general, 
long-term research into the design and impact issues relative 
to the evolution and operation of a national network. 
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A comparison of network architectures— 
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INTRODUCTION 

The ARPANET is a packet-switched data network which 
facilitates resource sharing and supports distributed data 
processing. The ARPANET technology and its descendants 
have had a significant impact on the direction of data com¬ 
munications over the past eight years. More recently, IBM 
announced Systems Network Architecture (SNA) which is 
targeted at satisfying a similar set of requirements. SNA will 
undoubtedly also have a major impact on the future of data 
communications. This paper presents a comparison of these 
two network concepts by discussing architectural and pro¬ 
tocol similarities and differences employed to support proc- 
ess-to-process level communication. The goal is both to 
provide useful information and to stimulate interest in the 
analysis of alternate architectural approaches. 

The presentation that follows identifies key properties and 
functions of data networks in general and discusses briefly 
how they are performed within each of the two network 
architectures. An elementary knowledge of both the AR¬ 
PANET and SNA is assumed. Information describing the 
design and implementation of the ARPANET can be found 
in References 1-3. A presentation of the concepts embodied 
within SNA can be found in References 4-8. 

Architectural similarities in these two designs result from 
fundamental similarities of objective. Both network archi¬ 
tectures have as primary goals the abilities to support dis¬ 
tributed processing and to facilitate resource sharing. In the 
case of the ARPANET the motivation for providing these 
capabilities was the need to share large hardware and soft¬ 
ware resources and to establish a facility to support com¬ 
puter system research. The development of SNA by IBM, 
on the other hand, was largely driven by the economic need 
to take advantage of equipment based on LSI technology. 
This technology provides the capability to locate many pre¬ 
viously centralized functions in terminals, device control¬ 
lers, communication front-ends, etc. Both designs have at¬ 
tempted to provide a single general-purpose interprocess 
communication facility upon which application level com¬ 
munication rests. A central theme in the development of 
both architectures is the separation of the data processing 
and data communication functions. 

Perhaps more important than the similarities of objective 


are a number of fundamental differences between the AR¬ 
PANET and SNA environments. These environmental dif¬ 
ferences provide a partial explanation of the individual ar¬ 
chitectural differences that are present in the two designs. 
The ARPANET was developed for operation within a re¬ 
search environment. Within such an environment it is pos¬ 
sible to experiment a bit in the design and operation of an 
evolving network. In a commercial environment, on the 
other hand, conservatism dictates more cautious develop¬ 
ment. While SNA has been developed as a new network 
architecture, it nevertheless has the requirement of accom¬ 
modating a large existing base of installed software systems 
(e.g., IMS, CICS). ARPANET development, on the other 
hand, was not constrained in this way. ARPANET host 
software was developed for the most part independent of 
any existing application software within the hosts. SNA was 
designed to provide a networking capability for equipment 
from a single vendor. The ARPANET, on the other hand, 
was developed from the start to provide a networking ca¬ 
pability for subscriber equipment from many different ven¬ 
dors. Finally, SNA and the ARPANET differ in the nature 
of the two network concepts. SNA is an implementation 
independent architecture. This architecture is realized dif¬ 
ferently within different IBM computer and communication 
products. The ARPANET, on the other hand, is an evolving 
implementation of the general concept of packet-switching. 

COMPARISON OF NETWORK CONCEPTS AND 

MECHANISMS 

The following subsections provide a side-by-side compar¬ 
ison of a number of aspects of the ARPANET and SNA 
network architectures. 

Physical network concept 

Figure 1 illustrates typical physical network configura¬ 
tions within SNA and the ARPANET. Figure la illustrates 
four distinct types of physical units (PUs) that can exist 
within an SNA network: PU.T5, PU.T4, PU.T2, and PU.T1. 
Units of type PU.T5 are host computers (370s). Communi¬ 
cation controllers (e.g., 3704s or 3705s) are described as 
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Figure 1—Physical network 


units of type PU.T4. Cluster controllers (e.g., 3601s) are 
described as units of type PU.T2. Finally, terminals (e.g., 
3270s) are units of type PU.T1. A basic concept of SNA is 
that there is only a single SNA node description and PU 
types are obtained as subsets of that single description. In 
this sense, PU.T1, PU.T2, and PU.T4 are subsets of PU.T5. 
The original announcement of SNA permitted configurations 
consisting of only a single host as shown by the solid lines. 
A more recent announcement from IBM has described ex¬ 
panded SNA implementations consisting of multi-host con¬ 
figurations (indicated by the dotted lines). 

Figure lb illustrates a typical physical network arrange¬ 
ment that may exist in the ARPANET. The two types of 
network nodes are referred to as IMPs and TIPs. The IMPs 
provide a store-and-forward switching function in addition 
to providing an interface for connecting host computers (H). 
The TIP is an IMP with an additional front-end concentrator 
function providing network access for unintelligent terminals 
(T). Arbitrary topological interconnections of IMPs and TIPs 
are permitted within the ARPANET design. A Network 
Control Center (NCC) provides a central facility for network 
monitoring, diagnosis and maintenance. 


Logical network concept 

Figure 2 contrasts the two network architectures with 
respect to their basic communication facilities. Within SNA 
the fundamental communication system is referred to as the 
transmission subsystem. The transmission subsystem pro¬ 
vides process-to-process level communication between ent¬ 
ities referred to as Network Addressable Units (NAUs). 
Three types of NAUs exist. The Logical Unit (LU) provides 
an interface port for SNA end-users. End users may be 
either human operators at a terminal or application pro¬ 
grams. Physical Units (PUs) comprise a second set of net¬ 
work addressable units. A distinct PU is associated with 
each shared communications resource (host, cluster con¬ 
troller, etc.) in the network. The PU is the entitiy to which 
communication is directed when one wants to communicate 
with the associated physical device. Finally, each SNA net¬ 
work has one (more in a multi-host network) System Ser¬ 
vices Control Point (SSCP). The SSCP provides central 
monitoring, coordination, and control of its domain in the 
SNA network. 

In the case of the ARPANET the IMP subnetwork con¬ 
sisting of interconnected IMPs (and TIPs) provides the basic 
communication facility. At this level the network hosts are 
the addressable units. It is important to point out that it is 
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the combination of the IMP subnetwork and the Network 
Control Program (NCP) software within the hosts which 
provides the process-to-process communication analogous 
to the SNA transmission subsystem. 

Although there are not any logical entities within the AR¬ 
PANET which correspond directly to the three types of 
NAU under SNA, certain analogies can be drawn between 
the two designs. Regular host computers (including the con¬ 
centrator pseudo-host associated with each TIP) are the 
most common type of host subscribers interfaced to the IMP 
subnetwork. The internal processes supported by these reg¬ 
ular hosts correspond to the logical units within SNA. Cer¬ 
tain fake hosts such as PARAMETER CHANGE and 
DEBUG, on the other hand, exist within each of the IMPs 
and provide a means of effecting changes in network oper¬ 
ation. They are, therefore, analogous to the physical units 
within the SNA design. Finally, the NCC facility can be 
viewed as analogous to the SSCP in the sense thqt it is the 
single network-wide coordination entity within the ARPA 
Network. A distinction between the SSCP and the NCC is 
the importance of the former for each conversation carried 
out over the network. In particular, it is directly involved in 
the initiation of LU-LU sessions as described in a later 
section. 

Figure 3 illustrates the types of communication between 
addressable units under the two network concepts. A line 
in this figure indicates communication between the entities 
corresponding to its end points. We have assumed in the 
case of SNA that only a single SSCP exists. If multiple 
SSCPs exist, then these entities will clearly require com¬ 
munication between one another in order to synchronize 
their activities. In addition, in the case of the ARPANET, 
we have neglected end-user to fake host communication 
which, while possible, is only incidental to the primary com¬ 
munication capabilities supported by the ARPANET. 


Key architectural characteristics 

Figure 4 summarizes a number of selected architectural 
characteristics in the case of the two network designs. The 
first entry in the table indicates that both SNA and ARPA¬ 
NET are based on block or packet-switching technology. In 
the case of the ARPANET, a packet has a fixed maximum 
number of bits. In the case of SNA, the maximum packet 
size is not only implementation-dependent but can also vary 
from node to node within the network. Under both designs 
packets are passed through the network according to a rout¬ 
ing algorithm. SNA has adopted a routing approach based 
on static tables within the nodes. The tables are centrally 
maintained by the SSCP. The ARPANET, on the other 
hand, has implemented a distributed adaptive routing algo¬ 
rithm in an attempt to dynamically adapt the flow of network 
traffic to changes in network topology and circuit loading. 
The routing algorithm is carried out periodically at each of 
the network nodes based on information that each network 
node receives from its neighbors describing the best path to 
each destination. 
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The processes of segmenting and reassembly provide the 
means for translating between data units (SNA PIUs and 
ARPANET messages) presented to the network and packets 
switched through the network. Under SNA, segmenting and 
reassembly can be performed at each node along the source- 
to-destination path. Within the ARPANET, packetizing is 
done only at the source IMP and reassembly is performed 
at the destination IMP after all of the packets of the message 
have arrived. Blocking and deblocking are associated with 
the combining of logically unrelated packets into a single 
block (superpacket) for efficiency of transmission over in- 
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dividual network links. This concept exists only within SNA 
where it is used for efficiency of transmission over 370 
channels. As indicated by the last entry in Figure 4, mes¬ 
sages (BIUs) and message fragments are kept in order over 
their source-to-destination path within SNA, whereas AR¬ 
PANET packets are routed independently and only reor¬ 
dered at the destination node. The ARPANET approach has 
potential benefits with respect to end-to-end delay and 
throughput but requires careful design of the control and 
buffering procedures in order to avoid the possibility of 
network lockups. 9 

The concepts of packetizing and blocking within the two 
network architectures are illustrated in more detail in Figure 
5. This figure illustrates the data envelopes associated with 
transmitting an application buffer of data from one process 
to another via the network. In the case of SNA the appli¬ 
cation buffer is first split into a number of Basic Information 
Units (BIUs), each consisting of a Request Header (RH) 
and a Request Unit (RU). A set of related request units is 
referred to as a BIU chain. (Note: A BIU chain can also be 


assembled from multiple application buffers.) The BIU chain 
is treated as a unit with respect to acknowledgment and 
recovery procedures. The Request Headers contain only 
control information to support process-to-process level com¬ 
munication. Application data is contained within the Re¬ 
quest* Unit portion of the BIU. Before a Request Unit can 
be transmitted, a Transmission Header (TH) is attached as 
shown. The Transmission Header contains end-to-end con¬ 
trol information such as destination address, sequence num¬ 
ber, etc. The combination of Transmission Header, Request 
Header, and Request Unit is referred to as a Path Infor¬ 
mation Unit or PIU. If required, a BIU can be segmented 
into BIU segments (each of which then becomes a PIU), as 
shown, prior to transmission over any network link. The 
PIU with Data Link Control (DLC) appended at the begin¬ 
ning and end is ready for transmission over the network link 
and is referred to as a Basic Link Unit (BLU). As illustrated 
in the last two lines of Figure 5a, independent PIUs can be 
combined or blocked at a node as described previously for 
purposes of transmission efficiency. 
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The situation is somewhat simpler in the case of the AR¬ 
PANET. Here an application buffer is split by the host into 
an arbitrary number of network messages, each message 
less than 8000 bits in length. Host-to-host protocol control 
information (H/H) is added and interpreted by the network 
control programs in each host in order to support process- 
to-process level communication. Each message is then di¬ 
vided into no more than eight packets of approximately 1000 
bits in length. These packets are routed independently 
through the IMP subnetwork. Host-to-IMP control infor¬ 
mation is provided by a leader attached to each message 
transmitted by a host. This leader is used to create the 
control information contained in the header of each of the 
data packets. 


Link control procedures 

Two types of link control are supported under SNA, the 
370 channel protocol and Synchronous Data Link Control 
(SDLC). 10 Analogous link control procedures supported in 
the ARPANET are the standard host-to-IMP interface, 11 the 
very distant host interface, 11 and the IMP-to-IMP protocol. 2 
The most appropriate of these protocols for purposes of 
comparison are SDLC and the ARPANET IMP-to-IMP pro¬ 
tocol. 

Although SDLC and the ARPANET IMP-to-IMP protocol 
both support communication over transmission circuits, 
these two protocols differ along several dimensions. SDLC 
implements data transparency by bit insertion in the data 
stream. Five sequential one bits in a row are always followed 
by a zero in the data stream to distinguish such patterns 
from framing patterns (which involve a sequence of six se¬ 
quential ones). Transparency in the ARPANET protocol is 
based on doubling of DLE characters. This procedure is 
similar to the one implemented under IBM’s older link pro¬ 
tocol, BSC, which was the basis of the IMP-to-IMP protocol 
frame structure. 

Another difference between SDLC and the protocol used 
on the links between IMPs is the diversity of the data links 
supported. Under SDLC a variety of data link configurations 
are supported, including point-to-point, multi-point, loop, 
etc. The ARPANET IMP-to-IMP protocol, on the other 
hand, is only designed for dedicated full-duplex point-to- 
point links. A third distinction between the two protocols is 
in the fundamental relation between the two ends of the 
link. Under SDLC each link has a primary and a secondary 
end. The primary end is typically associated with the station 
having superior processing capabilities. Under the ARPA¬ 
NET protocol both stations are assumed to have identical 
capabilities and the two ends of the link are symmetric. A 
minor difference between the two protocols is the length of 
the CRC checksum added for error detection. This check¬ 
sum is 16 bits in the case of SDLC and 24 bits in the case 
of the ARPANET. 

Perhaps the most interesting contrast between the two 
link control procedures is illustrated in Figure 6. As indi- 
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Figure 6—Link control 


cated, under SDLC the link control procedure implements 
what is fundamentally a single logical channel capable of 
supporting eight outstanding (unacknowledged) frames. 
These frames are numbered sequentially in order to enable 
the transmitting end to associate acknowledgments with 
data units. The analogous arrangement between IMPs is a 
sequence of eight logical channels each of which can support 
only one outstanding frame. In a sense these alternate ap¬ 
proaches are identical: if all frames are acknowledged prop¬ 
erly, both approaches support the same throughput over the 
link. If a frame is not acknowledged, however, the SDLC 
procedure requires retransmission of everything following 
the unacknowledged frame, whereas the IMP-to-IMP pro¬ 
tocol conserves channel bandwidth by requiring retransmis¬ 
sion of only the unacknowledged data. A more fundamental 
difference is a result of the dependency between frames 
which for SDLC is inherent in the sequential numbering 
procedure. In particular, failing to acknowledge any single 
frame completely blocks the flow of data over the link within 
eight frames of the frame which was unacknowledged. In 
the case of the ARPANET protocol, on the other hand, 
failing to acknowledge an individual frame transmitted over 
any logical channel does not impact the transmission of new 
frames over the remaining seven logical channels. 

The ARPANET IMPs take advantage of this independ¬ 
ence by permitting packet discard at the link control level. 
If a packet cannot be handled due to local congestion, the 
receiving IMP fails to acknowledge the packet as many times 
as necessary, being assured that its neighbor will continu¬ 
ously retransmit the data. Under SDLC this type of flow 
control cannot be carried out at the link level and an addi¬ 
tional higher level mechanism must be incorporated for this 
purpose. 
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Error control 

There are at least two types of errors which are associated 
with the transmission of data through packet switched net¬ 
works. Each type of error is typically handled by a distinct 
mechanism. Corrupted data due to burst noise on the com¬ 
munications channels or errors within the communications 
hardware is detected by appending checksums to transmit¬ 
ted data. Lost, duplicated, and out of order data is typically 
detected by the addition of sequence number information to 
each frame transmitted. These mechanisms can be applied 
both over individual links and over end-to-end paths. Figure 
7 illustrates the mechanisms for end-to-end error control 
which exist in the ARPANET and SNA designs. As shown 
in Figure 7a, there is a single end-to-end sequence number 
attached to each request unit transmitted by a process in an 
SNA network. This 16-bit sequence number is generated by 
the transmission control (TC) function of the source node. 
It is passed back to the source process in order that ac¬ 
knowledgment and error recovery information associated 
with that request unit can be identified. Errors due to cor¬ 
ruption of the data are handled by checksums on the links 
(associated with SDLC). 

In the ARPANET environment, sequencing is done on a 
host-to-host basis. Internal sequence numbers within the 
IMP subnetwork guarantee that messages are delivered to 
the destination host in the same order that they were sub¬ 
mitted by the source host. A 12-bit sequence number (“mes¬ 
sage I.D.”) field which is passed between the host and its 
IMP can be used for identification of responses (e.g., 
RFNMs) for multiple outstanding messages. It is also passed 
through to the destination host so that the destination can 
detect out of order errors due to subnetwork failures. As in 
the case of SNA, corruption of data on the lines is handled 
by link checksums. An additional end-to-end checksum is 
implemented in software in order to detect corruption of 
data originating within the nodes. This software checksum 
is computed on each packet by the IMP as it comes in from 
the source host and is checked at each node along the end- 
to-end path. 
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Flow control 

Figure 8 illustrates the type of process-to-process flow 
control implemented in the two network architectures. 
Under SNA, two logical units are always connected by two 
logical full-duplex flows, a normal flow and an expedited 
(priority) flow. The expedited flow is typically used for the 
transmission of process-to-process control messages. On the 
normal flow a form of flow control termed pacing is applied. 
Pacing messages sent from the receiver to the transmitter 
take the form of “Send N More RUs”. Such pacing requests 
can be sent after fewer than N RUs have been received in 
order to keep the end-to-end logical channel full in support 
of high throughput applications. 

In the ARPANET process-to-process communication typ¬ 
ically is carried out over a single pair of simplex channels. 
Control messages between all processes on two communi¬ 
cating hosts share a common control connection over which 
messages are handled with priority. Flow control is applied 
in both directions and allocations take the form of “Allocate 
M More Messages and B More Bits.” 12 Both M and B can 
vary from one allocate message to the next. 

Both of these flow control schemes can be described as 
incremental. This means that they both take the form of the 
receiver telling the transmitter that a certain number “more” 
data units are allowed. It has been argued 13 that such incre¬ 
mental flow control schemes have the potential for loss of 
synchrony if allocated messages are lost within the commu¬ 
nications system and no notification of this loss is provided. 
Windowing schemes tied more tightly to end-to-end se¬ 
quence numbers have been proposed recently. 14,15 Such win¬ 
dowing schemes have the property that they are self-syn¬ 
chronizing. 
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Addressing 

A major difference between process-to-process addressing 
in the SNA and ARPANET architectures is the use of net¬ 
work names for NAUs under SNA. Names or logical ad¬ 
dresses have the advantage of facilitating transparent move¬ 
ment of services between physical facilities by separating 
the concept of network name from the location of the named 
facility. Mapping between names and addresses under SNA 
is handled by the System Services Control Point during 
session initiation as described in the next section. Under the 
ARPANET design end-user identification is tied to physical 
location. 

The formats of the network addresses themselves provide 
another interesting contrast. SNA network addresses consist 
of 16 bits. The 16 bits are split between a sub-area address 
(used for store-and-forward routing) and an element within 
the sub-area (used for local routing). ARPANET network 
addresses consist of a 16-bit IMP number (used for store- 
and-forward routing), an 8-bit host number (used for local 
routing), and a 32-bit socket number (used for process iden¬ 
tification). In order to avoid having to transmit a pair of 32- 
bit socket identifications with each message transmitted be¬ 
tween processes, an 8-bit shorthand called the “link I.D.” 
is used in the ARPANET. This puts a (realistic) limit on the 
number of simultaneous conversations between any pair of 
hosts. 


Session initiation 

Figure 9 illustrates the procedures involved in setting up 
a logical session (connection) between two end users under 
the SNA and ARPANET architectures. The key point to 
note in this figure is that session initiation under SNA in¬ 
volves the SSCP as well as the two communicating pro¬ 



cesses, whereas session initiation within the ARPANET in¬ 
volves only the two communicating processes themselves 
(and their NCPs). Figure 9a illustrates the sequence of ex¬ 
changes assuming a primary LU wishes to establish a con¬ 
nection to a secondary LU. The primary LU first sends an 
INITIATE command to the SSCP. The INITIATE indicates 
the type of session requested, the name of the LU with 
which the primary wishes to speak, and password informa¬ 
tion for authenticating the request from the primary. If the 
establishment of the requested connection is permitted by 
the SSCP, it will respond to the primary by the transmission 
of a CONTROL INITIATE command. The CONTROL IN¬ 
ITIATE command contains a “bind image.” The bind image 
includes all of the parameters which are subsequently sent 
in a BIND command from the primary to the secondary LU. 
The parameters in the BIND command specify the type of 
session being established in detail. Assuming that the sec¬ 
ondary accepts the BIND request, the primary notifies the 
SSCP via a SESSION STARTED message. Session author¬ 
ization checks carried out by the SSCP are associated with 
the allocation of reusable resources as well as the name-to- 
address map. 

Communication between a user and a server process in 
the case of the ARPANET is established in one of two ways. 
For a service process which handles only one user process 
at a time, a simple pair of connection requests is exchanged. 
For multi-user server processes supporting multiple simul¬ 
taneous conversations, a standard procedure referred to as 
the Initial Connection Protocol (ICP) 16 is implemented for 
session initiation. The set of control messages which com¬ 
prise the ICP is illustrated in Figure 9b. The complexity of 
the required exchange is a result of the fact that connections 
are fundamentally simplex under the current ARPANET 
protocol standards (two are typically used for process-to- 
process communication), and the fact that two connections 
cannot have either of their ends in common. The ICP works 
by having the server process continually listening for con¬ 
nections from users on a “well-known socket” (L in the 
figure). A user wishing to obtain the service connects to this 
well-known socket and is subsequently switched by the 
server from the well-known socket to another pair of sockets 
which the server chooses. The server then continues to 
listen on socket L for connections from other remote users. 
In contrast to the tailoring of process-to-process communi¬ 
cation which under SNA is carried out as the result of the 
BIND exchange, tailoring of the session between user and 
server processes is a higher level protocol function. The 
ARPANET supports only a single type of process-to-process 
communication. 


Word length mismatch 

The issue of word length mismatch was of great concern 
to the designers of the ARPANET. This is due to the fact 
that the IMPs are 16-bit machines, and the hosts supported 
are heterogeneous. Attached host systems include IBM 370s 
(32-bit word length), PDP-lOs (36-bit word length), PDP-ls 


Figure 9—Session initiation 
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(18-bit word length), minicomputers (16-bit word length), 
and CDC6600s (60-bit word length). Word length mismatch 
is not so important an issue under SNA because equipment 
from only a single vendor is involved. In addition, most IBM 
mainframes and peripherals have byte handling facilities. 

In order to neutralize the problems of word length mis¬ 
match, the ARPANET designers provided two capabilities. 
The first enables the transmission of messages containing 
arbitrary numbers of bits (less than the maximum message 
size) through the network. The second facilitates the com¬ 
bining of multiple successive messages into a single appli¬ 
cation buffer without excessive bit shifting. The first goal 
was achieved by having the host/IMP interfaces add and 
interpret hardware padding sequences. The IMP interface at 
the source adds padding bits at the end of each message it 
receives from its host. The padding consists of a 1 followed 
by as many zeros as are required to fill out an IMP word. 
The host interface at the destination adds as many additional 
zero padding bits to the message as are required to fill out 
the word length of the receiving host. To avoid excessive 
bit shifting, the host-to-host protocol is defined to have a 
“message header.” The length of this header in conjunction 
with the leader associated with each message insures that 
the text portion of the message starts in a “good place” for 
most machine word lengths (i.e., on a word boundary). In 
addition, the host-to-host protocol specifies a “connection 
byte size” which can be used to guarantee that the message 
text also ends in a “good place.” 

SUMMARY AND CONCLUSION 

The previous sections have detailed and contrasted the 
mechanisms supporting a variety of network functions in 
the ARPANET and in SNA. An important similarity of 
approach is the hierarchical layering of functionality and 
protocols which is characterisitic of both designs. Commu¬ 
nication in both cases is characteristic of both designs. Com- 
nication in both cases is carried out between pairs of peer 
functional layers in the two communicating entities. This 
approach has advantages with respect to ease of compre¬ 
hension, maintenance, debugging, and expansion. Both de¬ 
signs are fundamentally packet switching technology and 
both designs involve some amount of physical separation as 
well as logical separation of data communication and data 
processing. 

More interesting than the conceptual similarities are the 
key architectural differences. These differences are more 
evident in SNA implementations than in the definition of the 
architecture itself. Centralization of network functionality is 
one such critical difference. Perhaps due to its ancestry, 
centralization is more clearly in evidence in the SNA design. 
A key example of this is the SSCP involvement in session 
initiation in single SSCP implementations. Symmetry of peer 
elements with respect to control is a second important dis¬ 
tinction. Two communicating entities in an SNA environ¬ 
ment often communicate as a primary to a secondary. One 
logical (physical) end of the communication path (link) is 
endowed with more control capability than the other. In the 


ARPANET, on the other hand, communication is typically 
symmetric (primary to primary). A third important distinc¬ 
tion between the two designs is the nature of the routing 
strategy. The distributed adaptive approach developed for 
the ARPANET is one of the key features of that design. 
While such routing strategies are not necessarily excluded 
from implementation under SNA, there does not appear to 
be a strong leaning in this direction on the part of the SNA 
designers. Finally, we point to general complexity as an area 
where the two architectures differ. The architecture of SNA 
is capable of supporting both half and full duplex links, 
switched and non-switched links, multipoint links, and loop 
configurations. SNA session protocols are tailored at the 
transmission subsystem level for efficiency in contrast to 
the single process level communication protocol within the 
ARPANET. In addition, a variety of network elements with 
different physical capabilities are accounted for under SNA, 
whereas only a single type of network element exists in the 
ARPANET design. 

The previous sections have attempted to present a side- 
by-side comparison of the ARPANET and SNA architec¬ 
tures. The goal of this comparison was not only to provide 
useful information but also to raise questions and spark 
some interest in comparing the relative merits of different 
architectural approaches. Moreover, by carefully examining 
two points in the space of possible architectural implemen¬ 
tations, it is possible to gain a better understanding of ad¬ 
ditional architectural alternatives. 
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INTRODUCTION 

A primary requirement for a message system in an opera¬ 
tional military environment is that it be secure enough to 
process messages at multiple levels of classification. But for 
the system to be accepted, the operator interface must be 
“usable.” Usability relates to three things: features pro¬ 
vided, ease of entering commands, and overall performance. 
Certainly an interface that does not perform well in these 
respects, or is difficult to learn or use, will not be used. 

The Department of Defense Advanced Research Projects 
Agency (DARPA) and the Navy are sponsors of an experi¬ 
ment to evaluate the operational use of a computer-aided 
message handling system at PACOM Headquarters in Ha¬ 
waii. The experiment will evaluate the operational and or¬ 
ganizational impact of the automated service on a commu¬ 
nity that now uses a largely manual system. The purpose of 
this paper is to document the security design of SIGMA,** 
the system that will be used in the experiment. 

In the following section, a description of the SIGMA mes¬ 
sage processing system is given. The third section provides 
background and discusses the kernel approach to multilevel 
security. In the fourth section we describe several security 
problems encountered in the design. The fifth section pre¬ 
sents the design of the SIGMA message service. The addi¬ 
tional features that the kernel must provide to support 
SIGMA efficiently are documented in the sixth section. Fi¬ 
nally, a summary is provided to highlight the paper’s main 
points. 

* This research was supported by the Defense Advanced Research Projects 
Agency under ARPA Order Nos. 3338, 2223 Contract Nos. F19628-78-C- 
0001, DAHC15-72-C-0308, MITRE Project No. 8010, ISI Project Nos. 3D30 
and 3P10. The views and conclusions contained in this paper are those of the 
authors and should not be interpreted as necessarily representing the official 
policies, either expressed or implied of the Defense Advanced Research 
Projects Agency or the United States Government. 

** The SIGMA message service is one of three services developed during 
this experiment. The other two are the HERMES system built by Bolt, 
Beranek, and Newman, Inc. 1 and DMS built by Massachusetts Institute of 
Technology 2 . 


THE SIGMA MESSAGE PROCESSING SYSTEM 

Information Science Institute of the University of South¬ 
ern California developed SIGMA specifically to meet the 
message handling needs of a military command. SIGMA is 
a secure interactive message handling system providing 
computer-aided message handling services for the receipt, 
filing, retrieval, creation, and coordination of military (AU¬ 
TODIN) messages. We consider it secure in that it presents 
an interface to the user constrained to abide by the DoD 
security policy. It is an interactive system since all user- 
system communications occur via an on-line terminal with 
a CRT display. Finally, it is a message handling system 
because it supports the typical message processing functions 
needed by any formal organizational operation. 

SIGMA supports the full cycle for processing incoming 
and outgoing messages in a military operation. It provides 
flexible filing capabilities for on-line storage of all messages. 
Easy access to messages and files is provided by selective 
search and retrieval functions. Incoming AUTODIN mes¬ 
sages move through the system by informal forwarding or 
by formal action assignment. Outgoing messages are proc¬ 
essed by a set of functions supporting message creation, 
editing, coordination, release, and post-transmission come¬ 
back copies. 

SIGMA operates on a DEC PDP-10 computer with the 
TENEX operating system. AUTODIN messages enter 
through the local AUTODIN message exchange. SIGMA 
distributes these messages to all pertinent addressees on the 
system where each user can access them through his SIGMA 
display terminal. 

THE KERNEL APPROACH TO MULTILEVEL 

SECURITY 

The need to process multiple levels of classified data led 
the Air Force Electronic Systems Division to sponsor sev¬ 
eral research and development efforts to build an operating 
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system that could satisfy by technical verification that DoD 
security requirements had been met. 3 Many of the results of 
the ESD work have been borrowed in the design of SIGMA, 
specifically adherence to a mathematical model based on the 
concept of a reference monitor—an abstract mechanism that 
controls the flow of information within a computer system 
by mediating every attempt by a subject (active system 
element) to access an object (information container).* 4 The 
hardware/software mechanism that implements the refer¬ 
ence monitor is called a security kernel. The kernel uses the 
rules of the mathematical model 5 , 6 as a specific policy for 
mediating access requests. This incorporation of policy into 
the kernel allows for a proof that verifies that the kernel 
correctly applies the policy to the information it protects. 7 , 8 

The mathematical model establishes an “inductive na¬ 
ture” of security by demonstrating that security is preserved 
from one state to another. Security is defined with two rules: 
the simple security condition and the *-property. 9 The for¬ 
mer states that a subject (active entity) cannot observe the 
contents of an object (information container) unless its se¬ 
curity level is greater than or equal to the security level of 
the object.** The latter further restricts possible access by 
stipulating that a subject may only modify an object if that 
object’s security level is greater than or equal to the security 
level of the subject. 

The purpose of the simple security condition is to prohibit 
users from obtaining data that they are not entitled to see. 
The *-property is designed to prohibit a program operating 
on behalf of a user from reducing the classification of any 
information. 

When a user is given a clearance, he is charged with 
responsibility for maintaining the classification of classified 
information. A computer utility cannot necessarily be given 
this same trust. This is due to: the amount of information 
that may be compromised; the speed with which the com¬ 
promise may occur; and the difficulty in detecting or appre¬ 
hending the violating program. By enforcing the *-property 
on computer programs, a program will not be able to either 
accidently or maliciously compromise information. Design¬ 
ers of computer utilities constrained by the *-property must 
ensure that *-property enforcement does not unnecessarily 
restrict the capabilities of the user. 

The enforcement of the *-property allows us to reduce the 
volume of code that needs to be trusted to a central section 
of the operating system. This central section of the operating 
system is the software component of the security kernel. To 
provide security, a kernel must (1) mediate every access by 
a subject to an object, (2) be protected from unauthorized 
modification, and (3) correctly perform its functions. A ker¬ 
nel satisfies the first requirement by creating an environment 
within which all non-kernel software is constrained to op¬ 
erate and by maintaining control over it. 

The requirement to protect against unauthorized modifi¬ 
cation is satisfied by isolating the security kernel software 


* In a computer system, subjects are users and processes, and objects 
include programs, data files, and peripheral devices. 

** Currently the security levels used in SIGMA are only the four standard 
DoD classifications, i.e.. Unclassified, Confidential, Secret, and Top Secret. 


in one or more protection domains, for example, by a ring 
mechanism. 10 Finally, the requirement that the kernel cor¬ 
rectly perform its functions is satisfied by using a formal 
methodology. A suitable methodology was introduced by 
Bell and Burke. 7 It includes: (1) a proof that the kernel 
behavior enforces the desired policy; 11 and (2) a proof that 
the kernel is correctly implemented with respect to the de¬ 
scription of its behavior used in the first step. 12 

We designed SIGMA with security kernel technology in 
mind. However, due to the absence of a kernel on the PDP- 
10 (the machine we used), the current implementation was 
done without a kernel. We have rigorously scrutinized the 
SIGMA design to ensure that the user interface provided 
would remain unchanged should SIGMA be reimplemented 
on a security kernel. In addition, the security primitives 
have been evaluated to ensure that their usefulness warrants 
their being included in a kernel. 

SECURITY PROBLEMS OF MESSAGE PROCESSING 

SYSTEMS 

A kernel supporting the current mathematical model of 
the DoD security policy is well suited for certain environ¬ 
ments, such as a programming environment in which users 
operate at a single security level for long periods of time. 13 
A message processing environment presents several prob¬ 
lems not found in previous environments, including (1) the 
dynamic nature of the user’s “working security level”; (2) 
the desire to present to the user information at more than a 
single security level; (3) the desire to accurately inform the 
user of the security level of all information he is reading and 
writing; and (4) the ability of users to extract text informa¬ 
tion and place it in a message of a lower classification than 
the source. 

The user’s “working security level” in a message system 
environment is considerably more dynamic than in the pro¬ 
gramming environment. Each time that a user performs an 
action on a different message, his working security level 
may have to change; for example, a user reading a Secret 
message may generate an Unclassified reply. While we could 
require the user to process messages at a single security 
level at a time, the resulting user interface would be clearly 
unacceptable to the user. 14 

To deal with these problems a new approach is needed 
that includes: a terminal that will allow users to process 
information at more than one security level at a time; and 
trusted processes that are able to violate the security rules 
in a controlled manner. The next section describes the se¬ 
curity architecture of the SIGMA message processing sys¬ 
tem. 

SECURE MESSAGE PROCESSING SYSTEM 

ARCHITECTURE 

The SIGMA security design has two goals: to produce a 
certifiably secure service, and to present the user with an 
agreeable user interface. In many situations these goals are 
at cross-purposes. Our general approach has been to present 
the user with a true picture of what is happening, maintain 
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Figure 1—Sigma process structure 


the user’s data at the proper level (or higher if this is not 
possible), and make it convenient for the user to do the right 
thing. 14 

Overview of the design 

When using SIGMA, a user is actually interacting with a 
collection of up to five processes (see Figure 1). These are 
the trusted process, an unclassified control process, and one 
process for each classified level that the user/terminal is 
cleared to operate. Each process (except the trusted proc¬ 
ess) can write data only at its own level and can read data 
at its level or lower. 

SIGMA attempts to be as helpful to the user as it can, 
organizing the user’s session and cleaning up the user’s state 
(current context) as necessary. The service also attempts to 
understand the user’s current context and conform its be¬ 
havior to the situation. For this reason the context infor¬ 
mation must be available to all user processes; thus, it must 
be unclassified. 

The user’s state in SIGMA is divided into two parts. The 
first part contains the current list of objects being accessed 
and functions being performed by the user. This portion of 
the state is maintained at the unclassified level by the un¬ 
classified control process. The second part contains the cur¬ 
rent list of message entries (from the open message file) in 
which the user has expressed an interest. The entry list 
information is potentially classified at the level of the file 
and is thus maintained at this classification level by the 
appropriate classified process. 

This dichotomy of state is reflected directly into the se¬ 
curity design. Commands are divided into those which ac¬ 
cess the unclassified state (unclassified commands) and 
those which access the entry list (classified commands). The 


latter group includes both commands that use the entry list 
for input and those that allow the user to enter classified 
information as part of the command. 

Multilevel terminal 

We designed the terminal, used by SIGMA, to enable the 
user to interact with data at more than one security level at 
a time. The screen of this “multilevel terminal” is divided 
into “windows” (see Figure 2), each of which is logically 
an independent terminal. Each window scrolls independ¬ 
ently and may have a different security level. Windows are 
further divided into domains that have various attributes 
(e.g., enterable, editable, underlined, etc.). The domain’s 
security level is the same as that of the window. 

To keep the user apprised of the level of information he 
is viewing and entering, we added two sets of lights to the 
terminal. Each set consists of four lights (one for each se¬ 
curity level); one and only one light of each set is on at any 
time. The first set is mounted on the keyboard; it specifies 
the classification of the window in which the cursor cur¬ 
rently resides. If the user wishes to know the level of any 
particular piece of information on the screen, he may move 
his cursor to the information. The second set of lights is 
mounted next to the screen and specifies the maximum level 
of information displayed on the screen. 

Form of command input 

We designed command input so that it could be done 
through a separate window that is normally at the unclas¬ 
sified level in order to keep the majority of the user’s state 
information at unclassified. Certain commands, such as the 
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Figure 2—Multi level terminal 


“find text string” command, have potentially classified ar¬ 
guments. For these commands the security level of the 
command window is raised to the level of the object that 
the command is affecting before the user enters the param¬ 
eters. 

Strict enforcement of the security model eliminates any 
possibility of a security compromise: a write-down path 
through the system that could be used to release information 
of a higher security level to a lower security level. 15 How¬ 
ever, even with the enforcement of the model, there are 
several situations in a message system where the user, by 
following instructions given by the system, can inadvertently 
compromise small amounts of information. 

Consider the following example: A user asks for a list of 
all his messages with a subject having word “x” in them. 
To perform this operation, the user must be at the security 
level of the file that he is looking at—greater than or equal 
to the security level of all the messages within that file. The 
enforcement of the *-property forces the result of this ex¬ 
amination to be at the level at which the examination was 
performed—the security level of the file. Should the user 
then decide to perform any modification to a message re¬ 
turned by this examination that has a security level lower 
than the file security level, the *-property would require him 
to: issue commands at the security level of the message that 
he desired to modify, and tell the system the unique iden¬ 
tification of the message told to him by the classified proc¬ 


ess. (The unique identification is required here because the 
system is unable to pass the desired identifier “down” due 
to the enforcement of the *-property.) However, this trans¬ 
mission through the user of the message-id from the higher 
process to the lower process is, itself, a violation of the 
*-property. Although it is conceivable that a maliciously 
written program could use this *-property violation to com¬ 
promise information, we assume that the user serves as an 
effective filter in this write-down path (both in “bandwidth” 
and in checking for reasonableness), thereby precluding any 
reasonable software means of making use of this path. 

Because of the hardships that strict enforcement of the 
*-property imposes on the user and because of the existence 
of *-property violations, a case can be made to ease the user 
interface in situations where this type of violation exists. 
The improvement takes the form of allowing SIGMA, in 
violation of the *-property rule of the security model, but 
with user concurrence, to write-down the unique identifier 
of the message that the user wants to modify. We limit the 
bandwidth of this type of *-property violation, so that it is 
no larger than the path that otherwise occurs through actions 
of the user, by allowing only a specific amount of fixed- 
format information to be transmitted to a lower security 
level and then only if the user has depressed an appropriate 
function key that is linked directly to the security kernel. 
Allowing the system to transmit this information greatly 
simplifies the user interface. 
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Trusted process functions 

Certain functions need special capabilities to operate 
(such as the passing of message identifiers) but are relatively 
message-system dependent and thus are not included in the 
security kernel. We group these functions together in a 
“trusted process” that has the ability to transfer information 
in a controlled fashion in violation of the *-property. 

The trusted process in SIGMA performs four functions: 
change classification; message release; command comple¬ 
tion signals; and entry list transmission. 

Change classification 

SIGMA allows users to change the classification of text 
that they are allowed to access. When this happens, the 
trusted process clears the screen and presents the text in a 
simple fashion (19 lines at a time) for confirmation. When 
the user has confirmed the entire object, the trusted process 
logs this action and passes the text to a process at the new 
security level for refiling. 

Message release 

In the military formal messages are released with the 
commander’s signature. Therefore, we consider the act of 
releasing a message a security event. To control message 
release, we require that the trusted process insure that the 
user requesting the release is authorized to release and that 
this user is making the request from an authorized terminal. 

An additional security consideration with message release 
is that some AUTODIN terminals (ours in particular) treat 
the message header as unclassified. In SIGMA this header 
is created in the same window as the message text. There¬ 
fore, releasing a message implicitly lowers the classification 
of the header information. During message release the 
trusted process requires the user to confirm that all of the 
header information is actually unclassified. The trusted 
process logs this action before releasing the message. 

Command completion signals 

We have based the SIGMA design on the concept of an 
unclassified process that receives the majority of the com¬ 
mands, determines the proper security level needed to ex¬ 
ecute these commands, and then activates a process at that 
security level to perform the execution. The disadvantage 
of this approach is that, should an error occur between the 
unclassified control process and the classified operational 
process, the classified process cannot ask for clarification. 
Thus error recovery is difficult. This problem is referred to 
as the open loop problem. 

Presently we believe that the best solution to the open 
loop problem is to allow the trusted process to close the 
loop when an error of this type is encountered, provided the 
user has depressed a function key since the last such re¬ 
quest. Closing the loop improves recovery but has an impact 


on security, since a *-property violation exists when the 
loop is closed. As in command input, requiring a user action 
between successive writedowns restricts the bandwidth of 
this operation. 

Entry lists 

When a user process needs to write down a list of message 
identifiers, “entry lists,” it passes this list to the trusted 
process for user confirmation directly. The trusted process 
checks the format and bounds of the entry numbers, and 
asks the user to directly confirm the number of entries being 
processed at each security level. Thus, the user has the 
ability to directly monitor (and control) the bandwidth of the 
writedown channel. This separate step is reasonable even 
from the user interface side, for if the number is too high or 
too low, the user can see that he specified his request in¬ 
correctly.* 

KERNEL REQUIREMENTS 

In order to be able to support the SIGMA architecture, 
the security kernel must provide certain additional features 
not found in kernels designed to date, including a terminal 
multiplexor for the multilevel terminal, a variety of object 
sizes, the ability to support large numbers of processes, an 
efficient inter-process communication facility, and a policy 
that can support “trusted” processes. 

Multilevel terminal certification 

Since the terminal supports the simultaneous display and 
editing of data at different classifications, we must demon¬ 
strate that the terminal (1) maintains the proper levels for 
all information it contains (possibly 20,000 characters) and 
(2) marks all information returned to the computer with the 
proper security level. It is the terminal’s responsibility to 
assure that no information entered in a window by either the 
user (doing local editing) or the application computer is 
transferred to any other window. While at first pass the 
certification of the terminal may seem trivial, one must con¬ 
sider that the terminal code is currently produced for a single 
INTEL 8080 and occupies 32K bytes of PROM. Eventually 
multiple 8080’s, application of Denning’s flow analysis, 16 or 
the introduction of a kernel in the terminal will be necessary 
to guarantee separation of the windows. 

Terminal multiplexor 

A significant problem is the method for attaching the mul¬ 
tilevel terminal to a secure system. We have identified two 


* If the entry list has only one element, then the appropriate function key is 
sufficient and the further “confirm 1 entry” step is omitted. This operation 
allows the user to point to a single entry or mention it by number or context 
(CURRENT, NEXT) for a display, reply, file, etc., without being required 
to do more than use the proper function key to enter or confirm the command. 
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alternatives: each window could have a unique connection 
to the system or the kernel could multiplex all information 
to a terminal over the same communication line. We have 
chosen the multiplexing approach in order to minimize the 
number of terminal lines. 

Communication between the system and the terminal is 
in the form of NOTICES and DISPATCHES.* The terminal 
multiplexor must assure that each NOTICE received from 
the terminal is directed to a user process whose security 
level is the same as the security level of the window. The 
multiplexor must also forward Function Key Notices to the 
trusted process to provide for the capability for a controlled 
write-down of message identifiers. 

The terminal multiplexor must insure the correctness of 
all DISPATCHES to the terminal. With the exception of 
“window allocation” DISPATCHES, the terminal multi¬ 
plexor need only check the window identifier and length to 
assure that the user process is communicating with a window 
to which it has access. All requests for terminal window 
allocations and deallocations must be done by the unclas¬ 
sified user control process. This process provides the ter¬ 
minal multiplexor with the security level for all newly cre¬ 
ated windows. The unclassified process can, at a later time, 
request a change in a window classification by notifying the 
multiplexor. If this new security level is lower than the 
current window security level, the multiplexor must erase 
the information currently in the window. 

Process structure 

The design of the kernel’s process structure will have 
significant implications for the performance of SIGMA. On 
traditional timesharing systems, such as TENEX, process 
creation is expensive, and process swapping is lengthy. In 
order for SIGMA to operate efficiently the kernel must be 
able to (1) support large numbers of processes; (2) allow for 
fast process creation and deletion; and (3) swap processes 
with little overhead. Large numbers of processes are re¬ 
quired because SIGMA requires several active processes 
per user. If SIGMA is extended to handle compartmented 
intelligence, fast process creation and deletion would be 
required. Finally, because large numbers of processes are 
doing small amounts of processing, process swapping occurs 
often. To expect a kernel to provide this type of support 
may require significant hardware support. 

Interprocess communication 

Equally important to the efficient operation of SIGMA are 
the speed and types of interprocess communication provided 
for by the kernel. SIGMA will require the kernel to support 
both preemptive (interrupt-like) and non-preemptive (mes¬ 
sage-like) types of interprocess communication. In addition, 
the latter mechanisms must support small messages for pass- 


* NOTICES and DISPATCHES are packets of information to and from the 
host computer respectively. 


ing message-ids and large messages for transmitting entire 
command strings. 

File system 

The file system is often one of the most complex portions 
of the kernel, a quality which can cause unnecessary over¬ 
head. For example, SIGMA does not require the kernel to 
provide a file organization such as a directory hierarchy. A 
“flat file” system is entirely adequate and can be used more 
efficiently. The only special requirements for SIGMA are 
that the kernel should support both small files (512 bytes) 
and large files (10K bytes). 

System integrity controls 

To support SIGMA the security kernel must provide a 
mechanism that implements the notion of least privilege. 
This mechanism has been given the name “System Integ¬ 
rity.” 17 SIGMA uses three separate privileges: a secure 
write-down privilege used by the trusted process to reclas¬ 
sify text; a release privilege used to restrict the releasing of 
messages to a select group; and a system security officer 
privilege used to initiate and set the security level of new 
users. 

The primary rule that the system integrity controls must 
obey is: to modify an object or execute a kernel call a 
subject’s system integrity level must be greater than or equal 
to the system integrity level of the object or kernel call. 18 
There are no rules on reading or executing programs (pro¬ 
grams in execution use the system integrity level of the 
process that they are executing under). We must therefore 
demonstrate that each subsystem with a system integrity 
level greater than system low (non-kernel, no privileges) 
does not execute any programs other than ones that we 
know execute properly. 

CONCLUSION 

The design of SIGMA demonstrates that it is possible to 
build a secure message processing system based on the 
kernel approach using the DoD security model. We have 
shown the refinements to the security policy that are needed 
to achieve a usable interface and have documented the fea¬ 
tures that a security kernel must provide to support a secure 
message processing system efficiently. The techniques used 
in designing SIGMA should be directly applicable to other 
transaction-oriented or data base management systems. 
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INTRODUCTION 

Networking affords a communications support mechanism 
among heterogeneous computers. It does not provide any 
means for alleviating the effects of differences across sys¬ 
tems. As a result, network users are forced to resolve such 
differences; the resulting learning and programming burden 
inhibits effective utilization of networking capabilities to 
achieve resource sharing. 

Achieving the potential of networking would be increased 
through masking system differences from the user in many, 
if not most, applications. Network Operating Systems 
(NOSs) 1 ’ 2 are commonly viewed as the mechanism for ac¬ 
complishing this objective. The functional objective of an 
NOS is to support and simplify access to existing services 
and to expedite the construction and subsequent accessing 
of new services by simplifying interaction among systems 
and between systems and users. 

The remainder of the Introduction discusses the required 
enduser support which must be provided by a Network 
Operating System, identifies the two major resulting imple¬ 
mentation issues and describes the constraints which affect 
the complexity of NOS implementation. Thereafter, the next 
section describes major NOS design decisions guiding the 
implementation of an Experimental NOS (XNOS) being im¬ 
plemented at the National Bureau of Standards (NBS) as 
part of a joint RADC/NBS effort. The third section then 
discusses the implementation of the user-system interface 
while the fourth section discusses the system-system inter¬ 
face. The fifth section provides some concluding remarks. 


Network operating systems 

A Network Operating System providing ease of access 
and utilization of systems, subnetworks, and services must 


* This work is a contribution of the National Bureau of Standards and is not 
subject to copyright. Partial funding for the preparation of this paper was 
provided by the U. S. Air Force Rome Air Development Center (RADC) 
under Contract No. F 30602-77-F-0068. Certain commercial products are 
identified in this paper in order to adequately specify the procedures being 
described. In no case does such identification imply recommendation or 
endorsement by the National Bureau of Standards, nor does it imply that the 
material identified is necessarily the best available for the purpose. 


provide two capabilities for endusers: job ececution and data 
handling. 

A network job consists of a collection of network job 
steps, each of which may be executed on a different ma¬ 
chine. The Network Job Execution function of a Network 
Operating System should provide the means for initiating 
job steps, migrating input information as appropriate, suit¬ 
ably disposing of output results and interacting with the user 
as required. 

Network Data Support provides the means for a user to 
manipulate files using a command language and for a pro¬ 
gram to access remote records at run time. To provide a 
homogeneous viewpoint across systems, the NOS must pro¬ 
vide a Network Wide Directory listing network accessible 
data resources, a common command language for manipu¬ 
lating files listed in this directory, and a means for limiting 
user access to these files to provide appropriate security 
mechanisms. At the record access level, mechanisms are 
required for preserving the meaning of data being transmit¬ 
ted between heterogeneous systems. 

Implementation of these two enduser functions requires 
two major support functions: user-system interfaces which 
provides a common virtual viewpoint for the user across 
heterogeneous systems and system-system interfaces to 
support the transmission of data between systems and pre¬ 
serve the meaning of the transmitted data. Such interfaces 
are discussed in sections three and four, respectively. Their 
implementation requires resolution of some underlying de¬ 
sign issues. 


Design issues and alternatives 

The difficulty of NOS implementation is critically de¬ 
pendent on the hardware and software environment which 
must be'accommodated. Operationally, such situations can 
be divided into three categories: (i) those requiring a retrofit 
of the NOS to a heterogeneous collection of computer sys¬ 
tems; (ii) those invovling retrofit to a homogeneous collec¬ 
tion of computer systems; and (iii) situations in which there 
is no constraint to interface to existing software. 

A study of these alternatives shows the following: (i) ret¬ 
rofitting to heterogeneous hosts is the most difficult case 
but maximizes resource sharing and best accommodates 
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both planned and existing systems; (ii) retrofitting to a ho¬ 
mogeneous collection of host computers is easier than ret¬ 
rofitting to a heterogeneous collection of hosts, but limits 
the opportunity for resource sharing—a major rationale for 
network construction; while (iii) unconstrained (e.g., built 
from “scratch”) implementations will be the most satisfac¬ 
tory from a user viewpoint, but lack downwards compati¬ 
bility and, as a result, will be extremely costly to implement 
since significant rewriting of existing software is required. 3 

NOS related projects 

Before beginning our discussion of XNOS, we first briefly 
describe three NOS related projects: RSEXEC, NSW and 
ACCAT. 

RSEXEC 4 was the first NOS oriented project. Since it 
was implemented for a homogeneous collection of computer 
systems (the TENEX systems within the ARPANET), the 
user-system interface problem was non-existent. Its primary 
contribution was exploration of the issues in providing a 
network wide file space and in resolving references to re¬ 
mote files. Such references were handed through an encap¬ 
sulation mechanism termed a JSYS trap 5 which, upon trap¬ 
ping such a reference, determined where the file was actually 
located and arranged its migration to the site of the executing 
program. 

The National Software Works 6 is designed to provide a 
sophisticated program production service. This is accom¬ 
plished through networking together some of the more so¬ 
phisticated program production tools available on different 
systems and, via NSW support mechanisms, making this 
collection appear as an entitiy to the individual user. Thus, 
upon issuance of a request to utilize a particular tool, the 
user and any appropriate files are automatically migrated to 
one of the sites at which the tool is available. As a result, 
the user only sees the provided tool and is unaware of the 
system on which it is being run. With the spectrum of ac¬ 
cessible objects thus constrained, it proves feasible to pro¬ 
vide a uniform treatment of error responses. 

The Advanced Command and Control Architectural 
Testbed (ACCAT) is an interesting study in the problems of 
interfacing a user to a Database Management System 
(DBMS). A major problem in such an interface is the re¬ 
quirement that the user learn the syntax and semantics of 
the query language. To circumvent this, ACCAT contains 
a restricted natural language interface which transforms the 
user’s request into an internal form. This form is then trans¬ 
lated into a series of queries against the database. 7,8 Collec¬ 
tively, this provides a fairly “soft” interface of the user to 
the DBMS. 

The preceding projects are primarily concerned with the 
utilization of networking capabilities to achieve specific 
functions. An explicit investigation of the interface issues 
implicit in supporting a general purpose NOS across heter¬ 
ogeneous hosts is being undertaken in the Experimental 
Network Operating Systems (XNOS) project at NBS. The 
functional components being investigated by these four proj¬ 
ects are illustrated in Figure 1. 
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Figure 1—Major NOS functional components 

NOS IMPLEMENTATION 

Implementation of a Network Operating System requires 
a series of design decisions. This section discusses two of 
the major configuration related decisions. The following two 
sections then articulate the issues in supporting the user- 
system and system-system interfaces and discuss implemen¬ 
tation specific factors. 

Support components 

Construction of a Network Operating System requires 
both the design of certain procedures and processes, as 
discussed in the following two sections, and determination 
of the processors on which they are to be executed. Perhaps 
the major design decision relates to where computing re¬ 
quirements are to be provided. Two alternatives exist for 
providing these requirements: the existing hosts within the 
network (the unaugmented case) or, alternatively, adding 
specialized processors to satisfy most of these computing 
requirements (the augmented case) which we will call Net¬ 
work Interface Machines (NIMs). 

Utilization of NIMs for satisfying most of the NOS com¬ 
puting requirements has four major advantages. First, it 
minimizes the impact of the NOS upon the host. This was 
deemed highly desirable since it is felt that few installation 
managers would tolerate any significant system overhead 
for NOS support. Even fewer would be willing to accept 
modifications of the host operating system. 

The second advantage to offloading NOS support is the 
consequent minimization of the need to recode functionally 
identical modules to accommodate different systems and 
architectures. This approach also facilitates centralized de¬ 
sign, implementation and support of the NOS. Hence, the 
third advantage—centralization—reduces costs by minimiz¬ 
ing the number of support personnel. 

The fourth advantage is facilitation of incremental expan¬ 
sion, modification and enhancement of NOS capabilities. 
This seems particularly important since evolving experience 
driven requirements are likely to result in gradual change of 
the initial NOS. 





Network Operating Systems 


775 


These four advantages must be considered in the context 
of two major implementation issues: attaching NIMs to hosts 
and the cost of providing the NIMs. After discussing the 
first issue and describing the initial XNOS implementation, 
we shall return to the second. 

Attaching NIMS to hosts 

Attaching a NIM to a host can be accomplished in two 
ways: implementation of the NIM as dedicated host(s) on 
the network or as a frontend processor to the host. To the 
authors, the second approach seems appropriate when Net¬ 
work Operating Systems have achieved a greater degree of 
maturity. A significant amount of work has been concerned 
with the relationship between the host and the frontend. 9 
The primary concern of those studies is with optimization 
of the provided functions rather than with identification of 
the functions to be supported. Since present research is 
more appropriately oriented toward identification, realiza¬ 
tion of the NIM as a dedicated host(s) on the network allows 
concentration on NOS support issues and therefore seems 
preferable. 

A major advantage of implementing the NIM as a dedi¬ 
cated computer attached to the subnetwork is the conse¬ 
quent utilization of the host-host protocol to support com¬ 
munication between NIM and hosts. The disadvantage is 
forgoing the simplification of connecting hosts which are ill 
suited to interactive use. 

The initial implementation—XNOS 

An Experimental Network Operating System (XNOS) has 
been implemented at the National Bureau of Standards to 
facilitate investigation of the interface issues (discussed in 
the following two sections) which must be supported to 
achieve more effective utilization of network accessible re¬ 
sources. Figure 2 illustrates the initial XNOS configuration 
and shows that NOS support is to be provided through a 
NIM attached to the communications subnetwork (the AR¬ 
PANET). 

Given that the NIM is to be a dedicated host on the 
communications subnetwork, only one is required to support 
a full feasibility demonstration. This follows since ARPA¬ 



NET technology permits a host (the NIM in this case) to 
establish a network connection with itself. Thus, a single 
NIM can provide the functions required to interface to a 
host issuing a request, the functions required to interface to 
the host responding to the request, and the communications 
functions required to interface between two NIMs. Thus, 
the basic implementation approach illustrated in Figure 2 
provides a reasonable, general purpose test environment. 

The preceding comments have provided a general per¬ 
spective on the adopted XNOS implementation approach. 
In this environment, the NIM is a PDP-11/45 running the 
UNIX operating system and attached to the ARPANET via 
a slightly modified version of the University of Illinois Net¬ 
work Control Program (NCP). 

The XNOS supported hosts are: a Honeywell 6180 run¬ 
ning the MULTICS operating system, a PDP-10 running 
TOPS-10, a PDP-10 running RENEX, and a PDP-11/45 run¬ 
ning UNIX. Based upon present experience we foresee little 
difficulty in supporting any interactive system attached to 
a communications subnetwork, preferrably using full duplex 
communications. Support of batch systems has not yet been 
investigated. 

Logging in 

Two basic approaches to logging into XNOS can be con¬ 
sidered. In the first approach the user is directly logged into 
a host computer and invokes XNOS support software 
thereby being indirectly logged into XNOS. In the second 
the user directly logs into the NIM (and hence directly into 
XNOS). The first approach potentially allows the network 
user to see the network as an extension of the host and to 
use the same command language used by that host com¬ 
puter. 

Adoption of the second approach seems more natural from 
an NOS viewpoint since it permits equal treatment of all 
hosts. Thus the user physically logs into the NIM, issues 
commands to the NIM receives responses from the NIM 
and, in general has the NIM acting as a mediator between 
the user issuing the requests and the systems satisfying 
them. This approach also simplifies support of utility func¬ 
tions. 

Utility support 

Effective support of network users is facilitated through 
the existence of certain support capabilities, e.g., editors, 
that function in an identical manner regardless of the host 
to which one is attached. XNOS supports this requirement 
by providing such utility functions on the NIM. Invoking 
these functions requires that any required input data or files 
also be located on the NIM. This is accomplished through 
using the mechanisms in sections three and four. 

NIM computational requirements and costs 

The cost of providing NIMs is dependent upon their com¬ 
putational requirements and the number provided. Such 
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costs must be carefully considered since the canonical prob¬ 
lem in providing any type of frontend support to a host 
computer is the tendency for such support to require another 
‘host’ of size comparable to that being supported. We feel 
that this is not a problem of XNOS since the only functions 
provided by the NIM are those supporting user-system and 
system-system interfacing together with utilities. 

Computational requirements 

The computational requirements for supporting user-sys¬ 
tem interfacing can be divided into: (i) support of the com¬ 
mon command language for file manipulation and the net¬ 
work job execution command, and (ii) support of NIM 
independent interactions. Typically, for an individual user, 
the average time between issuing commands in the first 
category would be on the order of minutes. The second 
category of interaction occurs when no intervention by the 
NIM is required, e.g., the user is interacting with a program, 
and the only computational support is that of forwarding 
commands or responses. We assume that the overhead in 
handling such commands is minimal. 

The discussion in section four shows that the computa¬ 
tional requirements for supporting system-system interfac¬ 
ing are those required to provide a network wide Interpro¬ 
cess Communication (IPC) mechanism and those required 
to support translation and transformation of records gener¬ 
ated by remote record accesses. 

IPC is used to support both user-system and system-sys¬ 
tem interfacing. As discussed above, the former is invoked 
relatively infrequently. We conjecture that the mean time 
between occurrences of remote record accesses by an in¬ 
dividual program will also be on the order of minutes due to 
the subnetwork delays which occur in accessing remote 
hosts. Moreover, computational support of the translation 
and transformation requirement generated by a remote rec¬ 
ord access is relatively minor as follows from the discussion 
in section four. (These estimates are only intended to indi¬ 
cate that NIM computational requirements are likely to 
prove acceptable. Their verification requires measurements 
which can only be performed after a significant level of user 
utilization of XNOS capabilities.) 

Utility support, particularly editing, can require a signifi¬ 
cant fraction of system resources. However, such support 
offloads the host of a function which it could provide (for 
interactive systems). Since this is done, in part, as a con¬ 
venience for the host, the requirement for supporting utility 
functions should not be directly counted against NIM com¬ 
putational requirements. 


Cost considerations 

The cost of providing NIMs is dependent upon the number 
supported and the size of each. Assuming that NIMs are 
implemented as hosts on the communications subnetwork, 
the minimum number of NIMs is one. A likely upper bound 
is one for each host. In an operational version of XNOS, 


the actual number used will probably be heavily influenced 
by a combination of performance considerations and the 
opportunity which NIMs afford for offloading certain utility 
functions (such as editing) from the hosts. A reasonable 
initial estimate for an operational environment would be one 
NIM per operating system class. 

Utilization of a NIM promises to become increasingly cost 
effective if declining hardware costs are balanced against 
increasing software implementation costs. Even now, a rel¬ 
atively powerful NIM can be procured for the equivalent of 
the burdened cost of supporting one professional program¬ 
mer. Moreover, since this approach minimizes reimplemen¬ 
tation of functionally similar modules and facilitates modi¬ 
fication, one must also balance the one time cost against the 
continuing programmer costs. 

Utilization of a NIM also permits offloading certain ter¬ 
minal support functions from the mainframe. As a result, 
system efficiency is potentially improved and systems which 
are ill suited to interactive utilization are made more usable 
in a networking environment. Of course, the relation be¬ 
tween the host and the NIM is an important factor in any 
final decision. 


THE USER-SYSTEM INTERFACE 

A Network Operating System should present a standard¬ 
ized view of network resources to its users and should pro¬ 
vide ease of access to those resources. This section de¬ 
scribes an implementation approach for a user-system 
interface that provides these functions for the XNOS. 

Implementation requirements 

The network user should be able to view network re¬ 
sources in the same manner that a user would view an 
individual system. A user knows of files in his workspace, 
but generally is not concerned with the complexities of the 
physical device upon which they are located. Likewise, the 
network user should be able to know there are files in his 
(network) workspace which exist on the available systems 
of the network, but should be insulated from the intricacies 
of the physical systems upon which the files reside. In ad¬ 
dition, since the network user would like to view the net¬ 
work as an entity, a common command language is desirable 
to support communication both with individual hosts and 
across hosts. 

These capabilities are provided within XNOS by a Net¬ 
workwide Directory System (NWDS) that describes the 
user, the user’s files and the available systems of the net¬ 
work. The NWDS also provides a common command lan¬ 
guage facility for manipulating known files and programs. 

The Networkwide Directory System (NWDS) 

The NWDS consists of (i) a data base, the Network Wide 
Directory (NWD), that describes the files belonging to each 
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user, the hosts on the network, and the types of interactions 
allowed among those entities; (ii) the software support re¬ 
quired to access the NWD data base; and (iii) the access 
control mechanisms that insure system integrity and data 
security. 

New users to the network are queried by the NWDS 
before entries are made into the NWD data base. These 
entries identify the user and the systems to be accessed. 
The user’s own network directory is then initialized and all 
connections to hosts are made on behalf of the user by the 
NWDS. Thus, the user sees the network as a large relatively 
integrated system composed of the resources available to 
the user on each of the accessed hosts. 

From the user’s viewpoint, files have no rigid naming 
conventions. To accommodate individual system con¬ 
straints, the NWDS maps these user file names into unique 
host file names conforming to the requirements of the host 
on which the file resides. Manipulation of these files is 
preformed by the NWDS for the user in accordance with 
user commands. 

In addition to providing a common command language for 
the user, the NWDS contains logical record descriptions and 
host representation tables to support preservation of mean¬ 
ing in transmitting records between heterogeneous hosts. 
This supports run time access by a program to records 
contained in remote files as discussed in the following sec¬ 
tion. 

User commands 

The NWDS provides a common command language for 
file manipulation and a command for executing network 
jobs. This section describes these commands and presents 
an example of Network Job Execution. 

Common command language for file manipulation 


XNOS file manipulation capabilities provide a common 
command structure for manipulating files residing on an 
individual system and for transferring files among systems. 
The chosen set of commands was developed by surveying 
file manipulation commands available on the systems cur¬ 
rently supported by XNOS. 

Determination of the precise set of commands to be sup¬ 
ported requires some care since support of a ‘network’ view¬ 
point of host resources is inconsistent with requiring detailed 
local host knowledge of how files are actually mapped to 
devices. As a result, only commands dealing with files as 
logical entities were selected for implementation as listed in 
Table I. Note that not all of the systems provided all of 
these commands. Thus, the NWDS effectively extends sys¬ 
tem capabilities. 

Network job execution 


A network job consists of a collection of network job 
steps. Each job step specifies a program, the host on which 


TABLE I—The Common Command Language 


File Handling Commands 


APPEND 

FILE1 FILE2 

COMPARE 

FILE1 FILE2 

COPY 

FILE1 FILE2 

CREATE 

FILE 

DELETE 

FILE1 

ERASE 


LIST 


RENAME 

FILE1 FILE2 

TYPE 

FILE 

UNDELETE 

FILE 

File migration Commands 

RETRIEVE 

FILE HOST 

TRANSFER 

FILE HOST 


it is to be executed, input data files and output data files. 
Prior to initiation of the job step all input files are checked 
to ensure that they are resident at the site where the program 
is to be executed. If not, appropriate transfer operations are 
begun and step execution is deferred pending completion of 
these transfers. The syntax for a job step is: 

RUN PROG@HOSTA 

IN 1=INPUT 1 ,IN2=INPUT2,. . . 

OUT 1=OUTPUT 1@ HOST 1 ,OUT2=OUTPUT2@ HOST2. 

where 

PROG is the program to be executed 

HOSTA is the execution site 

INn are the logical names for input data 

INPUTn are the names for the input data files 

OUTn are the logical names for output data 

OUTPUTn are the names for the output data files 

HOSTn are the final locations for the output data 

A network job and its job steps are defined by an inter¬ 
active program used in conjunction with the NWDS and the 
commands described above. The program queries the net¬ 
work user to gather all the information needed to specify a 
network job. For each job step in the network job, the user 
enters the program name, execution site, and for each input 
or output file, the logical name (name expected by the pro¬ 
gram) and the actual name (name in the user’s directory). 
The program builds a macro that invokes the system inter¬ 
actions necessary to perform the job step. The resulting 
command, called RUN, is issued like any other command 
and the network job is executed. Figure 3 indicates a sample 
session defining a network job. 


Access controls 

Acceptance of a Network Operating System requires ac¬ 
cess controls to limit the access of the user to programs and 
files and to limit the access of a program. User level access 
controls are provided through a profile capability which 
contains a complete description of the network user, the 
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The user wishes to execute the program TEST on the NBS PDP-10 
(TEST@NBS10). There are three input files, FILE1, FILE2 and FILE3 
which are to be given the logical names INI, IN2 and IN3 respectively. The 
program produces three output files: OUT1, OUT2 and OUT3. They are to 
be renamed and moved to other sites as follows: DATA1 goes to NBSUNIX, 
DATA2 goes to BBN and DAT A3 will remain at the NBS 10. There will be 

only one Network Job step in this job. A carriage return indicates no 
response. 

The following dialogue will take place between the user and the XNOS: 

Enter Program Specifications : TEST@NBS10 
Remote file access? (y or n) n 
Logical Name for Input File 1: INI 
Actual Name for Input File 1: FILE1 
Logical Name for Input File 2: IN2 
Actual Name for Input File 2: FILE2 
Logical Name for Input File 3: IN3 
Actual Name for Input File 3: FILE3 
Logical Name for Input File 4: 

Logical Name for Output File 1: OUT1 
Actual Name for Output File 1: DATA1 
HOST 1=NBSUNIX 
Logical Name for Output File 2: OUT2 
Actual Name for Output File 2: DATA2 
HOST 2-BBN 

Logical Name for Output File 3: OUT3 
Actual Name for Output File 3: DATA3 
HOST 3= 

Logical Name for Output File 4: 

Figure 3—Sample Definition of a Network Job 

systems he is permitted to access and the types of interac¬ 
tions allowed. 

Program access controls are supported in two ways: (i) 
run time checking of individual data items to ensure that the 
program has appropriate authorization, and (ii) execution of 
the program in its own run time directory to limit references 
by the program in a manner commensurate with the infor¬ 
mation provided by the user in issuing the job step execution 
command. 

In addition to these formal access control mechanisms 
some degree of additional protection is provided by having 
file names in the directories on individual hosts (Local Host 
Directories) which differ from the names by which the files 
are known to the user. Although the individual user will be 
permitted to determine host file names corresponding to 
those files in his directory under certain conditions, such 
information is not available for files belonging to other users. 

Support of such access controls requires that the NOS be 
the “owner” of all files resdient on an individual system. 
However, to facilitate interactions by a user wishing to di¬ 
rectly manipulate files, temporary directories can be set up 
which contain those files designated by the user and which 
support direct access by the user rather than access me¬ 
diated by XNOS. 


Implementation support 

The common command language described above was 
implemented using the capabilities provided by the Network 
Access Machine (NAM) 1041 developed at NBS. The NAM 
is implemented on a PDP-11/45 minicomputer running the 


UNIX operating system and provides directives which per¬ 
mit a user to interact with a remote system on a network. 
Sequences of these directives stored in files called macro 
files have been written that will generate the necessary ma¬ 
chine-dependent dialog to perform file manipulation func¬ 
tions required for each of the host systems. Thus, execution 
of each command in the language is actually a call to a 
macro file that the NAM expands and executes. 

The responses that the remote system generates are, in 
turn, analyzed by the NAM. By permitting the mapping of 
the varied responses from the remote systems into stand¬ 
ardized messages for the network user to see, the NAM 
facilitated the commonality of the common command lan¬ 
guage. The user can enter a standard set of commands and 
see a standard set of responses, and need not become fa¬ 
miliar with the responses of each system accessed. (Note 
that this approach supports standardized error messages 
across all anticipated error responses; unanticipated re¬ 
sponses are transmitted directly to the user.) 

SYSTEM-SYSTEM INTERFACING 

Successful system-system interfacing within a Network 
Operating System environment requires two primary func¬ 
tions: (1) a means for transmitting data between systems, 
and (2) a mechanism for accessing and preserving the mean¬ 
ing of structured data as it is transmitted across heteroge¬ 
neous systems. The first feature is provided by Interprocess 
Communication (IPC), and the second by a Remote Record 
Access (RRA) capability. 

In this section we shall discuss the problems of system- 
system interfacing in more detail and identify required func¬ 
tional capabilities and support mechanisms, while consid¬ 
ering some of the alternatives encountered during the im¬ 
plementation of XNOS. A more detailed treatment of the 
problem may be found in Reference 12. 

Interprocess communication 

Interprocess Communication (IPC) provides the basic 
mechanism for initiating and controlling the flow of data 
between cooperating processes. Since processes are the only 
active entities within a computer system, IPC is a basic 
building block for supporting communication between com¬ 
puters. 

The National Software Works 6 has implemented a rela¬ 
tively sophisticated IPC mechanism termed MSG 13 which 
permits the SENDing process to continue execution inde¬ 
pendently of the activities of the RECEIVEing process, 
whether or not the message was actually RECEIVEd. Pro¬ 
viding such a capability requires fairly complex software 
support. 

The XNOS IPC mechanism might be regarded as a mini¬ 
mum level IPC approach requiring very little host software 
to support its functioning. This is accomplished using a 
CALL/RETURN based mechanism that functions analo¬ 
gously to subroutine CALLs in that the WAIT state is en¬ 
tered upon issuance of the CALL. The simplicity of imple- 
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menting this approach stems from using constructs available 
in the major existing programming languages and requires 
minimal host support. 

The simplicity of XNOS IPC support derives from the 
fact that, if we assume interactive jobs, IPC messages can 
be transmitted over the controlling teletype channel, inter¬ 
cepted at the NIM and diverted from the user’s terminal to 
the appropriate destination where the reverse process oc¬ 
curs. Although the sophistication of this approach is limited 
and the bandwidth between communicating processes is 
constrained, the simplicity of its support coupled with the 
negligible startup costs for any host supporting interactive 
processing have led to its adoption. 

Remote record access 

Remote record access permits a process executing in one 
computer to access data records located in another computer 
at run time. This requires transmitting data between possibly 
dissimilar hosts which is complicated by both physical in¬ 
compatibilities between systems and organizational incom- 
pabitilities between the data as required/maintained by the 
systems. 

Physical incompatibilities include differences in word 
length, character packing, sign bit location, and data encod¬ 
ing (e.g., character sets or use of two’s vs. one’s comple¬ 
ment representation of numeric data). A translator is needed 
to convert data from the internal format of one host to that 
of another. Organizational incompatibilities between data 
bases, on the other hand, result in the need for a transformer 
to modify the format of files and data items (e.g., reorder 
and concatenate the data fields of a record before transmit¬ 
ting that data to the requesting host). 

The provision of a remote record access capability, then, 
requires: (1) a mechanism for selecting a record from the 
file/data base containing it, (2) a record translator to preserve 
meaning in transmitting the record between dissimilar hosts, 
and (3) a record transformer to permit the alternation of 
record structures. Supporting these functions mandates a 
means for describing the physical and organizational char¬ 
acteristics of the hosts. (It should be noted that the same 
functions could be performed iteratively for all records, 
when an entire file is migrated between dissimilar hosts— 
thus resulting in a file transfer protocol (FTP) for structured 
files.) 

These functional components will now be discussed in 
greater detail and, to better illustrate the associated infor¬ 
mation requirements, an example will be described in which 
a process on one host requests a data record from a dissim¬ 
ilar host on the network. As illustrated in Figuare 4, the 
host bearing the file containing the remote record is called 
the Data Host (Dhost) and that bearing the accessing pro¬ 
gram is the Process Host (Phost). 


Record selection 

The precise mechanism which supports record selection 
is dependent upon capabilities existing at the host computer 


including those provided by a data base management sys¬ 
tem, if any. The following discussion assumes the existence 
of a suitable mechanism for retrieving a record based on 
utilization of a unique key, if random access techniques are 
being employed or, alternatively, the keyword ‘next’ if se¬ 
quential access is being used. Given this assumption, we 
now describe the basic communication support. 

Assume that the Phost issues a request for a record lo¬ 
cated on the Dhost (path a[l] in Figure 4). This request, 
when received by the NIM, is forwarded to the Dhost (path 
a[2]. Figure 4), accompanied by any additional descriptive 
information needed by the selector. Such information would 
include the host specific file name, as determined by the 
Network Wide Directory System. 

Upon receipt of the request by the Dhost, a record selec¬ 
tor is responsible for retrieving the record and transmitting 
it to the record translator (path b[ 1], Figure 4). Figure 5 
shows such a record, in the format maintained by the Dhost. 
(The field names are included here for clarity, but would 
usually not be contained within the actual record.) Although 
the record in Figure 5 shows all data fields in character 
notation, it should be understood that these data items are 
actually in the machine representation of the appropriate 
data type (e.g., integer, real, logical, character, binary). 


Record translation 

Record translation preserves the logical record structure 
and data element type (e.g., real, binary, logical, integer, 
character) and, for arithmetic data elements, precision. Thus 
the record translator must know the exact format of the 
record to be translated down to the data item—a level of 
detail analogous to that contained in a FORTRAN format 
statement (e.g., 3I5,2X,5A5,F4.2). In addition, the internal 


I 

i 


I REMOTE RECORD ACCESS I 
I SUPPORT I 



l 


Dhost 


Phost 


DATA I 

BASE I 


a[1]: request for data from network-wide data base 
a[2]: request for data with access parameters 
b[lj: retrieved record in Dhost format 
b[2]: retrieved record in Phost format 

Figure 4 —Remote record access components 
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1 2 3 4 5 

12345678901234567890123456789012345678901234567890 


SSN NAME BIRTH ALRGYTEST HT 

OAOOOOIOOO 

333775555CHARLES J SMITHSON 10269200200F100030182 
WT SDISEASEl DATE MEDICATION DISEASE 

88075MMALARIA 0601943QUININE GOUT 

2 DATE MEDICATION DISEASE3 DATE 

0801965CORTISONE STREP 11039 

MEDICATION DISEASE4 DATE MEDICATION 

69PENICILLIN ANGINA PECTORIS0703972DIGITALIS 
MOTHER ADDRESS 

REBECCA B SMITHSON BOX 444, TAMPA FLA 00000 
FATHER ADDRESS 

FREDERIC SMITHSON DECEASED 
YCYMDFILLER 
1020T 

Figure 5—Data host record—DDR[D]* 


*The characters in the allergy-related fields are in hexadecimal representa¬ 
tion. The upper digit represents the first four bits of an eight bit binary field, 
while the lower digit represents the remaining four bits of that field. 

format of all data types must be known for each computer 
that is supported on the network. 

Placement of the record translator at the host requires its 
coding for each separate host architecture and increases the 
impact of the network on the host. For both of these reasons, 
the record translator has been implemented on the NIM. 
Although knowledge of the internal data representations of 
N hosts are still required, only 2N, rather than N(N-l) trans¬ 
lators must be implemented (one into and one out of each 
host format). Since host dependencies are reflected only in 
the tables being used to drive the translation process, 2N 
becomes 2. As the process of translating into and out of 
host dependent representations are very similar, it was ini¬ 
tially conjectured that 2 would become 1. However, there 
prove to be enough subtle differences to justify two separate 
programs rather than consolidating them into one. 

Given that translation is to be supported on the NIM, the 
remaining issue is whether to map directly from source to 
destination form or to proceed via a common intermediate 
form. The substantial simplification in implementation af¬ 
forded by the intermediate Network Normal Form (NNF) 
resulted in the selection of this approach. Figure 6 shows 
the Dhost record as it would appear after translation to NNF 
format. (Notice that all data types are now in a character 
representation.) Once the record has been transformed to 
meet the needs of the requesting process (Figure 7), it is 
again translated—this time into the bit representation re¬ 
quired by the Phost, a process that is essentially the inverse 
of the original, table-driven translation from Dhost format 


1 2 3 4 5 

12345678901234567890123456789012345678901234567890 


333775555/CHARLES J SMITHSON/1026920/+000000001010 
0010000000000000000000001111/+00000001000100000000 
00000000000000000011/+182.88/+75/M/MALARIA/0601943 
/QUININE/GOUT/0801965/CORTISONE/STREP/1103969/PEN1 
CILLIN/ANGINA PECTORIS/0703972/DIGITALIS/REBECCA B 
SMITHSON/BOX 444, TAMPA FLA 00000/FREDERIC SMITHS 
ON/DECEASED/+10/+20/*T*/ 

Figure 6—Data host record DDR[NNF] 

to NNF. Figure 8 shows the record in the Phost format. The 
resulting record is then transmitted to the Phost (path b[2], 
Figure 4) and RETURNed to the requesting process. 

The communications subnet often supports character 
translation. Therefore it would at first seem reasonable to 
have the translation of character data handled by the subnet, 
thus reducing the translation requirements to a minimum. 
However, as it is unreasonable to assume the exclusive use 
of character-oriented data, such subnetwork capabilities 
cannot be relied upon for the general case. It may be desir¬ 
able, however, to add to XNOS a capability to default to 
such subnetwork handling of data when dealing with char¬ 
acter-only data. 

Record transformation 

Record transformation supports modification of both the 
logical structure of the record and individual data elements. 
The record transformation function matches the information 
transmitted to the needs of the receiver or to ensure proper 
protection of sensitive information (e.g., matching the data 
to the access rights of the receiver by omitting sensitive 
information from the record before transmitting it on to the 
requesting process). Such transformation affects the logical 
structure of the record through one of three basic transfor¬ 
mation types: logical, arithmetic, or string. 

Logical transformations such as AND or OR generate 
Boolean binary strings resulting from the bit by bit ANDing 
and ORing of two successive strings. Arithmetic transfor¬ 
mations such as: +,-,/,x act as would be expected. String 
transformations can be quite complex as evidenced by the 

1 2 3 4 5 

12345678901234567890123456789012345678901234567890 

333775555/CHARLES J SMITHSON/M/+50/1026920/+000000 
0000000000000000000000000000000011/+00000001101100 
10000000000000000000001111/+165/+6.l/STREP/1103969 
/PENICILLIN/ANGINA PECTORIS/O703972/DIGITALIS/*T*/ 
Figure 7—Phost data record—PDR[PNNF] 
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1 2 3 4 5 

12345678901234567890123456789012345678901234567890 


ISSN NAME SYGBIRTH TALRGSALRGW 

| - 

I OOOOOOBOOO 

I333775555CHARLES J SMITHSON M301026920000031200F1 

|T HT DISEASEl TREATMENT DATE DISEASE2 

| - 

I65604STREP PENICILLIN 1103969ANGINA P 

| TREATMENT DATE DFILLER 

| - 

IECTORISDIGITALIS 0703972T 

Figure 8—Phost data record—PDR[P] 


capabilities of string manipulation languages. Initially, a con¬ 
catenation capability seems desirable. Among the additional 
transformations that may be needed are algorithms for the 
compression and/or decompression of textual information. 14 

As with the record translator, support of record transfor¬ 
mation is implemented more efficiently on the NIM. This 
follows through having the transformer operate on the rec¬ 
ord in NNF format. 

In XNOS a Transformation Description Table is used to 
provide the information that the translator needs to establish 
relationships between data items in the Phost and Dhost 
records. In this table each item in the Phost data record 
(PDR[i]) is expressed as a function of the appropriate item(s) 
in the Dhost data record (DDR[j]). For example, the con¬ 
catenation of two character fields from the Dhost record 
might be expressed as follows: 

PDR[j]=DDR[i] | DDR[k] 

A record that could be the result of such a series of opera¬ 
tions is shown in Figure 7. 

Supporting transformation of numeric data types (i.e., 
integer, real) requires care since precision must be consid¬ 
ered. Our approach uses a character based intermediate 
representation coupled with an arbitrary precision arithmetic 
package. The character representation directly accommo¬ 
dates varying precision while the arbitrary precision package 
supports transformations of numeric types. Of course, the 
precision of the result will never be better than the lesser of 
that of the source and destination. However, this approach 
does ensure that precision is not lost during the translation/ 
transformation process. 

Secure environments 

The enhanced access to system resources provided by a 
Network Operating System requires a suitable collection of 
access control mechanisms. Not only must a user’s identity 
be verified at login time, but access on his behalf to NOS 
maintained accounts on remote systems and to files, rec¬ 
ords, and fields within those systems must also be con¬ 
trolled. 

As part of a continuation of the joint NBS/RADC effort 


which resulted in the work described in this paper, efforts 
are currently under way to structure NOS access control 
requirements and alternatives. This study will result in the 
incorporation of prototype access control components into 
XNOS. Since the password has been found to be a relatively 
inexpensive, but highly effective mechanism for identify ver¬ 
ification and access control when used properly 15,16 it will 
no doubt be an integral part of the effort to enhance XNOS 
security. 

Once the decision has been made to control access to 
network resources, the ability to protect data during trans¬ 
mission becomes a further requirement. Although it is gen¬ 
erally accepted that an algorithm such as the Data Encryp¬ 
tion Standard 17 would be suitable for such purposes, an 
appropriate encryption key management mechanism would 
have to be determined for the NOS environment. These and 
other network security related issues will be addressed in 
subsequent studies and reports. 

CONCLUDING REMARKS 

In this paper we have provided a perspective on the issues 
and implementation constraints implicit in providing a gen¬ 
eral purpose Network Operating System for a heterogeneous 
collection of host computers. One implementation of an 
NOS has been described based on utilization of Network 
Interface Machines to offload NOS support from the host 
and to facilitate centralized design, implementation and sup¬ 
port. 

Based upon our experience in the implementation of 
XNOS we are led to conclude that NOSs are feasible. How¬ 
ever, in view of the complexity of the possible error mes¬ 
sages, the activities of the general purpose user of such a 
capability will probably be constrained to minimize unex¬ 
pected error responses. Thus, we would anticipate that gen¬ 
eral purpose computing might be confined to one or a few 
machines while other systems would be accessed to use 
unique resources known to be fairly well debugged. 

Finally, it should be noted that Network Operating Sys¬ 
tems only solve part of the problem of simplifying user 
access to systems—that of providing a uniform viewpoint 
across systems. The remaining part is concerned with sim¬ 
plifying system access given such a capability and consti¬ 
tutes the area of network access support. In Reference 18 
it is shown that such support can be structured into four 
major components: user and service profiles to simplify and 
automate user interaction with services; soft user interfaces 
to eliminate the need for user concern with the precise 
syntactic structure of commands; expert assistance to re¬ 
duce the need for entering repetitive information; and dy¬ 
namic tutorial assistance to provide specific, pinpointed 
problem solving information online. 
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INTRODUCTION 

With the advent of microcomputers and computer networks, 
distributed processing (i.e., the sharing of computing by 
several processors with each processor assigned to perform 
a certain task) becomes technically and economically fea¬ 
sible. Such architecture also provides system expandability, 
system reconfiguration, and fault tolerant capability for the 
system. The performance of such a system is not only af¬ 
fected by its application but also by the system architecture. 

One of the important problems that affects system per¬ 
formance is interprocess communication. It is intimately 
related with the task partition and assignment of various 
process modules and the system bus structure that provides 
the interprocess communications. In this paper we will first 
describe a new system architecture which consists of mod¬ 
ules with identical bus interface connected by a unique 
broadcasting bus structure. In order to reduce bus interface 
cost, serial broadcasting busses are used. Next, we describe 
the simulation model for studying the performance of the 
proposed architecture. Finally, we use the known program 
characteristics and workload profile of a Naval data com¬ 
munication network as an example of the proposed archi¬ 
tecture in order to study its feasibility and performance. The 
quantitative relationships among process and bus utilization, 
message delay, and task partitioning and assignments of the 
distributed processing system are presented. These results 
provide us with insight and understanding of the behavior 
of the distributed processing system. 

THE DISTRIBUTED PROCESSING SYSTEM 
ARCHITECTURE 

The proposed distributed processing system as shown in 
Figure 1 consists of three module types: processor, memory 


* This research is supported by the U.S. Office of Naval Research, Contract 
no. N00014-75-C-0650. 


and input/output (I/O) modules. Modules may communicate 
with each other via a unique broadcasting bus structure. 
Each module has an identical bus interface consisting of a 
serial send bus and several receive busses. The send bus 
transmits data serially to all other modules in the system. 
The receiver receives data serially on its receive busses. For 
such a bus organization, a system with N modules requires 
N busses to communicate among the modules. 

Every module has an identical hardware bus interface (BI) 
and a front end processor (FEP). The FEP handles data 
transfers and protocol between the bus and the module, 
coordinates data transfers, discriminates between message 
types, and handles bus transactions for all modules. The 
FEP of a processor module acts as an input/output controller 
for the main processor. When it is implemented as a micro¬ 
processor or as a bit-slice processor, it provides a reliable 
and inexpensive approach to handling data transmission and 
bus protocol. It also provides an inherent distributed system 
intelligence which can be programmed to control the allo¬ 
cation of hardware resources. A microprocessor is used as 
the main processor of the processor module. 

Data enters the system via the I/O module. The I/O mod¬ 
ule may store the data temporarily in its own FEP local 
memory, or it may transfer data directly to any other module 
via the bus. Similarly, the FEP of a processor module may 
store data received from the system bus temporarily in its 
own local memory or transfer it directly to the main pro¬ 
cessor local memory. The FEP of the memory module may 
accept data temporarily in its own local memory or transfer 
data to the common bulk memory. Any module can initiate 
system output by transferring data to an I/O module. 

The processor module 

The processor module organization shown in Figure 2 
assumes a microprocessor or bit slice processor implemen¬ 
tation. The module consists of a main processor, an FEP, 
local memories, a Direct Memory Access (DMA) facility, a 
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PROCESSOR MODULE MEMORY MOOUlE 1/0 MODULE 



Figure 1—A distributed processing system architecture 


bidirectional processor to processor interface and a system 
bus interface. The main processor executes the application 
program. The main processor local memory consists of read 
only memory (ROM) for program storage and random access 
memory (RAM) for data storage. The FEP has a small 
amount of local memory programmed to handle bus I/O and 



Transmitter 


Figure 2—An organization of the processor module 


bus protocol. Local memory is dedicated to each main pro¬ 
cessor for program storage to alleviate contention by several 
processors accessing common memory, thereby reducing 
the inter-module communications. The FEP transfers data 
to and from the bus, as relayed to it by the system bus 
interface, and writes the data in the main processor local 
memory through the direct memory access. The reverse 
process occurs when data is read from the main processor 
local memory. 

The DMA facility allows the FEP to store data in and 
retrieve from the RAM of the main processor. This interface 
ensures that data transfers are transparent to the interrupt 
and I/O control hardware of the main processor. The pro¬ 
cessor to processor communication interface allows word 
transfers to take place directly between the main processor 
and FEP without both accessing the main processor local 
memory. Either the main or the FEP can interrupt the other 
directly through their respective I/O facilities. The FEP re¬ 
ceives or transmits data from or to system busses and pro¬ 
vides asynchronous communications to the main processor, 
I/O control facilities, or to the main processor local memory. 


The broadcasting bus structure 

The proposed system may be viewed as a network con¬ 
sisting of different types of modules each having its own 
dedicated send bus which broadcasts information to all other 
modules. Further, each module can “listen” to the infor¬ 
mation from all the other modules in the system. Bus band¬ 
width affects the delay in inter-module communications; a 
serial broadcasting bus structure is used to both increase the 
bandwidth and reduce the bus and bus interface complexi¬ 
ties. 

Further, faults generated by hardware failures in such bus 
organizations are not likely to affect more than one bus, 
thus providing a graceful system degradation if software 
allows. The proposed bus structure can also be arranged to 
transmit parallel data thereby increasing the bandwidth for 
inter-module communications. However, system bus relia¬ 
bility decreases as the bus hardware complexity increases, 
and increasing the number of bus lines also increases the 
costs of the system bus interface. Hardware allocation 
schemes for bus hierarchies inherently become more com¬ 
plex as the number of busses increases. For real time ap¬ 
plications, contention for a bus among modules is a severe 
problem when the contention time exceeds a specified time 
requirement for the system to respond to an external inter¬ 
rupt. Broadcasting bus organizations alleviate bus conten¬ 
tion problems. 

For the proposed bus organization, a system with N mod¬ 
ules requires N busses to communicate among the modules. 
The system bus interface of each modules consists of N 
individual bus interfaces. Each bus interface (one per bus) 
decodes control signals from the FEP, performs serial-to- 
parallel data conversion, encodes and decodes character 
parity checks, and stores data for transmit or receive. Com¬ 
mercially available LSI components such as Universal Syn¬ 
chronous/Asynchronous Receive Transmitter (USART) or 
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Intel microcomputer (e.g. 8048) may be used for such inter¬ 
faces. 

Inter-module communications 

Communication between modules is initiated by a module 
(called the source) transmitting a header, a character, or a 
block of data at a time. The bus interface of each receiving 
module of the system receives and temporarily stores the 
character of data and interrupts its FEP. The FEP of the 
receiving module honors the interrupt by resetting the in¬ 
terrupt facilities in the bus interface and then reading the 
identification of the bus request. The FEP of the receiver 
reads and decodes the destination address. The source be¬ 
gins transmitting data after receipt of an acknowledgment 
from each of the designated receivers. The receiving FEP 
disables the bus communication by not resetting the inter¬ 
rupt facility if the data is not addressed to it or if it does not 
desire to receive the data at that time. The receiving FEP 
records the time when the transfer request character was 
recognized. Since a receiver must send an acknowledgment 
honoring the transfer data request within a specified “time 
out” period, the source waits to receive an acknowledgment 
before it begins sending its block of data. All data commu¬ 
nication on the bus is in block format. Asynchronous data 
communication requires DATA and ATTENTION lines for 
each bus. Synchronous data communication requires 
DATA, CLOCK, and ATTENTION lines for each bus (Fig¬ 
ure 2). The receiving FEP takes one character of data at a 
time from the bus interface and stores the data temporarily 
in its own local memory or stores the data, via the DMA 
interface, directly at a predetermined memory location in 
the main processor local memory. After a receiver acknowl¬ 
edges the header character (which may consist of destination 
ID, word count, priority and message types), additional 
characters are transmitted by the source as specified by the 
word count of each message block. 

In order to transfer data from the main processor to the 
bus output the main processor interrupts the FEP via the 
processor to processor interface. The data transfer flag for 
each receiver is stored in the RAM of the main processor. 
After the FEP acknowledges the interrupt, it reads the data 
transfer table if there is a request for a data transfer. The 
FEP initiates the output of that data block to the system bus 
interface. Transferring input data to its main processor is 
accomplished in a similar manner. Since program and local 
variables are stored in the local memory of each main pro¬ 
cessor, interprocess communication is greatly reduced. 


Input/output module and memory module 

Both the I/O module and the memory module (Figure 1) 
have an FEP for handling data transfers to and from the 
bus, and have identical system bus interface, FEP, and local 
memory. The memory interface in the memory module per¬ 
forms the function of interfacing the FEP with the bulk 
memory. The channel interface in the I/O module asyn¬ 


chronously multiplexes the channels and provides asyn¬ 
chronous communication with its FEP, I/O control, and 
interrupt facilities. 

The I/O module interfaces the system with the user and 
with a variety of peripheral devices that include various 
hardware interfaces. The I/O module stores the data in its 
own local memory which queues the data for output to an 
I/O channel interface. In a similar manner, the FEP retrieves 
data from its local memory and sends the data to the bus. 
The interconnection of the FEP of the I/O module to the 
channel interface depends on the specific application and 
channel interface characteristics. 

PERFORMANCE EVALUATION 

In examining the performance of a distributed computer 
system, we are interested in the study of inter-module com¬ 
munications, processor and bus utilization, and response 
time and throughput of the system. Since these perform¬ 
ances are program and application dependent, analytical 
models that can accurately predict the system performance 
are difficult to devise. An alternative method is to construct 
a prototype and measure these performance values. This 
approach not only requires the development of hardware 
prototypes, software system, data reduction and measure¬ 
ment facilities, but also encounters difficulties in evaluation 
of the system performance at different parameter values. 
We therefore use simulation techniques which enable us to 
use the actual program behavior and workload profile in 
studying system performance. 


The simulation model 

In order to provide a general yet accurate simulation 
model for our study the model is based on events. Events 
are generated at each processing module according to a 
given arrival distribution and enter into multi-level queues 
waiting their turn to be processed. The processing time of 
each task is known. In evaluating the performance of the 
system we are only interested in the task processing se¬ 
quence, the task time and the bus occupancy time. The 
information of processor instructions and bus traffic con¬ 
tents is not required. As a result, the model does not require 
the detailed knowledge necessary for the applications pro¬ 
gram and system software. This greatly simplifies the sim¬ 
ulation. The accuracy of the result depends on the accuracy 
of the estimates of processing times, bus occupancy times 
of various tasks, and the workload profile. 

The simulation model for a module is shown in Figure 3. 
The bus occupancy time of a task can be estimated from the 
number of bits transmitted and the bus bandwidth. All the 
queues are of a multi-level queueing system, with entry level 
determined by the priority of the messages. The model also 
has facilities to gradually increase the message priority as a 
function of the amount of time since it entered the system. 
The simulation is event-driven. That is, it advances the clock 
to the next event which is to occur, rather than stepping 
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Figure 3—Simulation model for a module 


through every clock interval and checking for events. Thus 
the time required for a simulation run depends on the num¬ 
ber of events which occur during the simulated interval 
rather than the time span of the simulated interval. The 
internal unit of time is expressed in terms of bus clock time. 
Inputs and outputs are represented in milliseconds. 

A module can receive data simultaneously from several 
busses, but can send data only on its designated send bus. 
The number of received busses, the number of modules in 
the system, as well as the buffer space can be easily varied 
in the simulation. 

Input parameters for the simulation are message arrival 
rate, which corresponds to the satellite communications 
channel bandwidth, effective processor execution time, 
which reflects both instruction execution time and the power 
of the instruction set, bus bandwidth, assignment of proc¬ 
essing tasks to the modules, and number of processing mod¬ 
ules. 

The performance measures used in the simulation are 
utilization of the processors, busses, and peripheral devices, 
delays at each module such as processing queues, sending 
queues to the bus, receiving queues from busses, and mes¬ 
sage response time. 

The simulation program creates a message control block 
for each network message to be handled by the system (send 
or receive). Each block is composed of sub-blocks which 
spepify the details of processor and bus usage and contain 
time stamps applied at the times of queue entry and depar¬ 
ture. After the processing of the message has been com¬ 
pleted, the time stamps in the message control block are 
used to calculate the delay statistics. 


The simulation is written in PL/C, Cornell University’s 
compile-and-go version of PL/1. Because subroutine calls 
have a high executed time overhead, no subroutines are 
used. Integer variables were used whenever possible in 
order to save execution time. The processing time required 
on the IBM 360/91 for a typical simulation run ranges from 
one to two minutes. 

EXAMPLE FOR SIMULATION 

The application selected for simulations was a communi¬ 
cations processor (at a ground station) in a military satellite- 
based communications network. This is mainly because rea¬ 
sonable estimates of the computational requirements, work¬ 
load profile, as well as program structure (program size, 
execution sequence and times) were available from a pre¬ 
vious study that was implemented with a military computer 
(AN/UYK-20). 1,2 

The communications network shares a single half-duplex 
channel of a satellite in stationary earth orbit. Reservation 
time division multiplexing (statistical multiplexing) 3,4 is used 
for multiplexing the communications from all the stations. 
There are two types of stations: master station and slave 
station. The master station not only handles all the slave 
station tasks but also it handles time slot reservation for all 
the slave stations. In our investigation we shall study the 
performance of the communication processor in a master 
station. The basic time unit (time slot) is the time required 
to transmit a 4,164 bit block. These time slots are grouped 
into a frame which consists of 15 to 32 consecutive time 
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slots. The first time slot of each frame is for reservation of 
the remaining time slots in that frame. The master station 
makes the time slot assignments on the basis of requests 
received from all the other stations (slave stations). These 
requests, which indicate the priority and number of mes¬ 
sages a slave station is waiting to transmit, are sent during 
designated assignment request time slots. An assignment 
permits a designated station to broadcast a message block 
to all other stations. The stations addressed by the header 
(part of the 4,164 bits) need to fully process the message. 
For our application, we always keep the channel busy. 
Therefore, dummy messages are transmitted (assigned by 
the master station) when there are not enough real messages 
to occupy all the available time slots. The amount of work 
involved in handling these dummy messages by the com¬ 
munication processor at the received station is similar to the 
handling of those messages that are forwarded to the other 
stations. Each block of data consists of a 16 bit cyclic re¬ 
dundancy check code. Error detection and retransmission 
are used for error control. 

Assumptions used in the simulation example 

In the simulation example, we assume the round trip prop¬ 
agation delay between the radio (ground station) to the sat¬ 
ellite is 260 msec. All the peripheral devices are assumed to 
have constant service times: Disk=90 msec; Printer=200 
msec (at 6000 lines/min.); Magnetic tape drive=150 msec. 
Processor execution time ratio is relative to a typical military 


minicomputer (AN/UYK-20) with instruction execution time 
of 1.5-2.3 msec. Execution time includes the effects of both 
instruction execution cycle time and the power of the in¬ 
struction set. Each module has 16K bytes of buffer space 
available for bus traffic and can receive data from three 
busses (i.e., three other modules) simultaneously. A mes¬ 
sage fits exactly into one message block (4,160 bits including 
message header and cyclic redundancy check bits). Proc¬ 
essing of a send message has a higher priority than the 
processing of a received message. This quarantees that send 
processing suffers minimum of delay, which assures that 
messages to be sent will be ready for transmission in their 
assigned time slots (as required by the network protocol) 
with a reasonable processing scheduling scheme. The bus 
bandwidth is assumed to be 50 kHz/bus. 

Task partition and assignment for the example 

The two main tasks of the communications processor at 
the master station are to perform send and receive message 
blocks. Their task flow sequence is shown in Figures 4 and 
5. Additional tasks such as generation of network and station 
status reports for the operator, creation of checkpoint tapes 
for system recovery in case of failures, etc. occur relatively 
infrequently. For purposes of simplicity they are not in¬ 
cluded in the simulation. 

The basic principle of a good assignment of processing 
tasks to processing modules is to balance the processing 
load and minimize the amount of inter-module communica- 


a) SEND A TIME SLOT RESERVATION MESSAGE 



b) SEND A TEXT MESSAGE 



Figure 4—Sequence of processing tasks for handling a send message 
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b) OUTPUT MESSAGE TO OPERATOR 



Figure 5—Sequence of processing tasks for handling a receive message 


tion. Assigning all the tasks to one module requires the least 
amount of inter-module communication but may result in 
long queueing delays due to processor overload. On the 
other hand, assigning each task to a module yields short 
queueing delay for the processors but requires a large 
amount of time for sending intermediate results between 
modules. Because of the inter-module communication, proc¬ 
essing requirements are rather complicated and may vary 
from one application to another. We shall use the simulation 
model to compare the performance of two task assignments. 

The first assignment consists of assigning the tasks to a 
five-processor module system as shown in Figure 6. The 
task for sending a message on the communication network 
is largely independent of the task for receiving a message 
from the network. To minimize inter-module communication 
between the send task and the receive task, they are as¬ 
signed to separate processor modules. We assigned one 
module for status record updating and time slot reservations, 
one module to serve as peripheral device controllers, and 
one module for file management and system recovery. 

The second assignment consists of assigning the tasks to 
a three-processor module system as shown in Figure 7. Both 
sending and receiving processing require access to status 
records, but usually there are more receive messages than 


send messages. In order to minimize the amount of inter¬ 
module communication, status record updating and time slot 
reservation operations are combined with the receive pro¬ 
cessor. System recovery and report generations are com¬ 
bined with the send processor. The file manager and disk 
controller are combined, along with the peripheral device 
controllers, into a single module. 

Table I displays the performance comparison of the three- 
processor and five-processor module systems. We notice 


TABLE I.—Performance Comparison between the Three-Processor Module 
and the Five-Processor Module Systems 


Number of processor modules 

5 

3 

Virtual time to process a received message 
(msec) 

1601 

1510 

Average queueing delay to receive a message 
(msec) 

14 

13 

Total # of bits transmitted on buses to process 
one received message 

13504 

8736 


The system is operated at the following condition: 59 percent of the received 
messages are addressed to the master station and require complete process¬ 
ing; Network data rate=9600 bits/sec.; Bus bandwidth=50kHz; processor 
speed equal to a minicomputer. 
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that the average delay in processing received messages is 
almost the same in both of these configurations. Due to less 
inter-module communication in the three-processor module 
system, less bus traffic is required to transmit the interme¬ 
diate results and less processing time is required to prepare 


and interpret these inter-module communications. As a re¬ 
sult, the three-processor module system yields better per¬ 
formance than that of the five-processor module system. 

It is interesting to note that due to the excessive inter¬ 
module communication and the overhead in handling these 



Figure 6—A five-processor module organization for the example. 
M=Memory requirements for program storage (in bytes) 
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TABLE II.—Processor and Bus Utilization Comparison between the Three- 
Processor Module and the Five-Processor Module System 


Utilization 

Processor 

Bus 

Number of processors 

5 

3 

5 3 


Percentage 

Percentage 

Receive processor 

5 

5 

9 9 

Status tables 

2 


1 

Send processor (SP) 

4 

4 

7 7 

Output queues and 

1 


1 

printer, tape controller 


5 

6 

File manager and 
disk controller 

5 


15 


inter-module communications in the five-processor module 
configuration, the throughput as well as response time of 
the three-processor module are better than that of the five- 
processor module system. Therefore, in the planning of a 
distributed computer system, using more processors does 
not necessarily always yield better throughput. In fact, one 
of the important problems in the planning of a distributed 



Figure 7—A three-processor module organization for the example. 
SP=Send processor, RP=Receive processor, RI=Radio interface. 
M=Memory requirement for program storage (in bytes) 



9.6 19.2 25 

NETWORK DATA RATE (10 3 BITS/SEC.) 


Figure 8—Average delay to process a receive message. Note that the sum of 
the delay components is greater than the total delay because some delays 
occur at the same time. Relative processor execution time=l. Bus clock 
period—20 msec (50 kHz/bus) 


processing system is how to partition the tasks and optimally 
assign them to the processor modules such that they yield 
minimum interprocess communications and high system 
throughput, and yet satisfy the cost and response time re¬ 
quirements. 

Discussion of results 

Total delay is defined as the time difference for processing 
a message with and without any queueing delay. Total delay 
is not necessarily equal to the sum of all the queueing delays 
encountered in the processing of a message, since some 
processing (and delays) occur in parallel execution branches. 
Delays are shown only for the processing of received mes¬ 
sages, since the processing of send messages has a higher 
delay allowance than that of the receive messages. Since the 
message arrival and departure process are deterministic and 
there is a long radio propagation delay, there is no interfer¬ 
ence between network cycles.* And since the processing of 
received messages is completed before the processing of 
send messages begins, we only need to simulate one com¬ 
plete network cycle. We studied the simulation variations of 
the network input data rate and relative processor execution 
time with the three-processor module configuration. The 
system performance (with processor execution time equiv- 


* A network cycle consists of 32 time slots: 1 for time slot assignments by 
master station, 9 slots for sending messages, and 22 slots for receiving mes¬ 
sages. 
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Figure 9—Peripheral device delay for processing a receive message vs. net¬ 
work data rate (10 3 bits/sec). Note: mag. tape drive delay is not included in 
total because it always occurs at the same time but is shorter than the printer 
delay. Relative processor execution time=l. Bus clock period=20 msec (50 
kHz/bus) 


alent to a minicomputer) for different network input data 
rate are portrayed in Figures 8, 9 and 10. From Figure 8 we 
notice that the largest component of total delay is due to 
queueing at the peripheral devices, to which the disk con¬ 
tributes the largest amount (Figure 9). The disk delay is 
large because it has a high utilization. From Figure 10 we 
notice that the bus utilization is only 5 percent at input rate 
equal to 9,600 bit/sec. Thus the bus delays (Figure 8) are 
small compared to an average bus occupancy time* of 44 
msec. From Figure 10 we noticed that the bus receiver 
utilization is lower than the bus utilization because the bus 
receiver utilization is computed on the basis of each module 
being able to receive from up to 3 busses simultaneously. 
Since no module is addressed by more than three other 
modules, there is no queueing delay for bus receivers. Due 
to the low processor utilization (Figure 11), the processor 
delay is very small compared to the longest uninterrupted 
processing time of 38 msec for the SP module and 13 msec 
for the PP module. Next we studied the performance as a 
function of relative processor execution time. A network 
data rate of 19,200 bits/sec was used. The simulation results 
are portrayed in Figures 12, 13 and 14. 


* Based on approximately equal numbers of large blocks of 4,192 bits each 
(about 85 msec) and small blocks of 192 bits each (about 4 msec) are sent on 
the busses during the processing of messages. 



9.6 19.2 25 

NETWORK DATA RATE (10 3 BITS/SEC) 

Figure 10—Utilization of busses and bus receivers vs. network input data 
rate. The bus transmitter utilization is the same as the bus utilization. Bus 
clock period=20 msec (50 kHz/bus) 


For the relative processor exe cution t ime exceeding 5 
(Intel 8080 has a relative processor execution time of ap¬ 
proximately 4 in this application), the total delay for proc¬ 
essing a receive message exceeded the maximum allowable 
time which is 218 msec for 19,200 bit/sec and 436 msec for 
9,600 bits/sec. From Figure 12 we noticed that for the system 
with large relative processor execution times, the delay of 
a received message is dominated by the processor delay. 
The increase in link queueing delay with increasing proces¬ 
sor execution times is due to the increases in the effective 
peripheral device service resulting from the reduction in 
processor speed. The decrease in disk queueing delay at 
high processor execution times is due to the stretchout of 



Figure 11—Processor utilization (for both send and receive messages) vs. 
network data rate. Relative processor execution time=l. 
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Figure 12—Average delay to process a receive message vs. relative processor 
execution time. Note that the sum of components delay is greater than total 
delay because some delays to a message occur at the same time. Network 
data rate =19,200 bits/sec. Bus clock period=20 msec (50 kHz/bus). The 
relative processor execution time for a minicomputer^, Intel 8080 type 
processor=4. 



Figure 13—Average delay at the processor module for processing a receive 
message vs. relative processor execution time. Network data rate=19,200 
bits/sec. Bus clock period=20 msec (50 kHz/bus) 


access request arrivals resulting from the increased proces¬ 
sor queueing delay. The variation in bus delay to received 
messages is the result of specific timing relationships be¬ 
tween the processing of send and receive messages that 
occur at certain processor rates, which is due to the varia¬ 
tions of starting time for processing of send messages. The 
queueing delay at the Receive Processor for processing a 
receive message is approximately equal to that of a PP 
module (i.e., balanced delay). The delay from the PP module 
is due to the larger number of processing tasks, each of 
shorter duration that for the SP module. Processor utiliza¬ 
tion (Figure 14) is the overall utilization average for the 
whole network cycle. When the processor execution time is 
large, the demand for processing may be higher than 100 
percent during part of the time. This yields a very large 
queueing delay. 

With the three-processor module configuration with Intel 
8080 type microprocessors, the system performance is ad¬ 
equate for handling of the example network with a data input 
rate of 9,600 bits/sec. Since three-processor modules are just 
adequate, it is very unlikely that a system with a two-pro¬ 
cessor module configuration could do the job satisfactorily. 
The response time for the two-processor module is likely to 
be much larger because the processing tasks cannot be easily 
divided into two equal parts. Consequently, additional de¬ 
lays will be caused by increases in contention between pro¬ 
cessors and by unbalanced processor loading. 

Simulation results reveal that the serial broadcasting bus 
structure is viable. The bus delays for receive messages are 
lower than the processor delays for a relative processor 
execution time greater than 3 and are comparable to the 
processor delays for a relative execution time in the ranges 
between 1 to 3 (Figure 12). Bus delay can be reduced con- 



J_I_1_1_L 
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RELATIVE PROCESSOR EXECUTION TIME 


Figure 14 —Processor and peripheral devices utilization (includes processing 
of both send and receive messages) vs. relative processor execution time. 
Network data rate—19,200 bits/sec. 
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siderably with a higher bandwidth bus. A much higher bus 
bandwidth interface with 2.5 MHz (50 times higher than the 
bus bandwidth used in the simulation) is available commer¬ 
cially. 

We also noticed that increasing the network data rate 
causes a higher bus and processor utilization, while increas¬ 
ing relative processor execution times increases only the 
processor utilization. 

CONCLUSIONS 

Simulation study revealed that the proposed distributed 
processing architecture with three processor modules pro¬ 
vides adequate processing capability for handling the Naval 
data communication network processing tasks. Further, the 
serial broadcasting bus structure provides sufficient bus 
bandwidth for the application. 

We also found that task partition and task assignment to 
various processor modules has great influence on interpro¬ 
cess or intermodule communications and thus affect the 
system throughput. A good rule for optimal task partition 
and assignment is to balance the workload among the var¬ 
ious modules and yet keep them as independent as possible 
so as to reduce inter-module communication. Excess inter¬ 
module communication not only requires excess bus band¬ 
width but also requires extra processing time for handling 
these inter-module communications. This degrades system 
throughput. 

The low bus and processor utilization for the system in¬ 
dicates that our proposed distributed processing architecture 
would be useful for other applications as well. Distributed 


processing architecture not only is more flexible for system 
reconfiguration and system expansion, but it also provides 
greater fault tolerant capability. Because of these capabili¬ 
ties and its low implementation cost, distributed processing 
architecture should find its place in many future applica¬ 
tions. 
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INTRODUCTION 

During the next decade, in the wave of microelectronics and 
digital revolution, we can expect to witness fast growth in 
the number of communication-based distributed data proc¬ 
essing systems. A communication-based distributed data 
processing system is a decentralized data processing system 
in which data processing facilities or computer resources are 
geographically distributed and communicate with each other 
through a data/computer communications network. Figure 
1 typifies such an environment. The most common data/ 
computer communication network architectures used in the 
distributed environment are those illustrated in Figures 2, 3 
and 4. These are the tree-shaped hierarchical, the packet 
and the ring-shaped architectures, respectively. These ar¬ 
chitectures are not new and heuristic algorithms have been 
widely available for their optimization. However, the em¬ 
phasis on the distributed data processing systems shifts the 
application environment and objectives of the optimization 
problem and hence lessens the applicability of these earlier 
algorithms. This paper presents a set of integrated algo¬ 
rithms more appropriate for network optimization in this 
new environment. Some of the algorithms are newly devel¬ 
oped; some are modifications of the earlier ones. 

Figure 2 depicts the tree-shaped, hierarchical architecture, 
in which the local access network defined as the circuits 
interconnecting the terminals and/or terminal control units, 
are connected to, and controlled by, the concentrating de¬ 
vices. The concentrating devices are in turn connected to, 
and controlled by, the host computer. The concentrating 
functions may be performed as part of a remote processor’s 
responsibility. The circuits shown interconnecting the con¬ 
centrating devices and the host computers are called the 
backbone network. Both the local access and the backbone 
networks may be point-to-point or tree-shaped connections. 
Occasionally, ring-shaped terminal connections may also 
occur. 

The mesh-shaped packet switching architecture is illus¬ 
trated in Figure 3. In this architecture the local access net¬ 
work comprises both the terminal devices and the host 


computers connected in either a point-to-point or multipoint 
configuration. The local access networks are connected to 
the packet switches. The backbone network for this archi¬ 
tecture is defined as the interconnecting circuits between 
the packet switches. 

Figure 4 presents the architecture associated with a ring- 
shaped, ring-switching design. The local access network in 
this case is the same as that defined earlier for the packet 
network architecture. Similarly, the backbone network is 
defined by the circuits interconnecting the switches, in this 
case specified as ring switches. For completeness, it should 
be noted at this point that terminals controlled by a Terminal 
Control Unit (TCU) usually form, on a second order local 
level, a star or ring network as shown in Figure 5. 

As evident from the figures, interfacing the local access 
and backbone networks are the concentrating devices, the 
packet switches, and the ring switches. In this paper, these 
devices will be generically referred to as the Network Ac¬ 
cess Facilities, or NAF’s. (For more detailed descriptions 
of the data network architectures, see Reference 1.) 

The network optimization problem of a distributed sys¬ 
tem, therefore, consists of the determination of the number 
and locations for NAF’s; the configuration of the local ac¬ 
cess networks, including point-to-point, tree-shaped multi¬ 
point and ring-shaped multipoint; and the configuration of 
the backbone network, including point-to-point, tree-shaped 
multipoint, ring-shaped multipoint, and mesh-shaped packet 
switching architectures. Of course, the optimization of each 
of these network problems is not new, and good heuristic 
algorithms have been widely used in many data/computer 
communication networks. However, new emphasis on the 
distributed data processing systems has lessened the appli¬ 
cability of these algorithms. The following sections of this 
paper discuss the algorithms and implementation approaches 
that are felt to be more suitable to the new distributed 
processing network environment. 

For local access networks with geographically dispersed 
terminal devices, “effective” algorithms for optimization 
have existed for over a decade. 2 * 3 The term “effective” 
implies nearness to the theoretic optimum. With the number 
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Figure 1—Communication based distributed data processing system 
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Figure 3—A packet switching architecture 
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Figure 2—Tree-shaped hierarchically controlled data communication 
network architecture 


of terminals on the order of hundreds or even thousands, 
the efficiency in terms of computer memory and running 
time requirements for executing the algorithms becomes im¬ 
portant. Otherwise, however effective, implementation of 
the algorithms may become too expensive. The second sec¬ 
tion describes a currently most effective algorithm for the 
optimization of tree-shaped multipoint networks and shows 
how it can be modified to improve its implementation effi¬ 
ciency. 

The concept of ring-switching has been proved workable 
for quite a few years, 4 and there are ring-shaped local access 
networks in operation; e.g., IBM 3600. However, with few 
exceptions, devices on a same ring are generally located in 
the same proximity. Therefore, topological optimization has 
never been a concern and there has not been direct published 
work in this area. Nevertheless, it is expected in the distrib¬ 
uted data processing environment that in some applications, 
the ring network may be more desirable than the tree net¬ 
work even for geographically dispersed devices. The third 
section describes algorithms for designing ring-shaped net¬ 
works, based on the modifications of graph-theoretic re¬ 
sults. 5,6 

Dominating the backbone network optimization problem 
is that of the packet switching network. The bulk of the 
research in the optimization of packet switching networks 
has been supported by the Advanced Research Project 
Agency from 1969 through 1976. Naturally, the algorithm 
that has been widely used and has been verified by real 
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Figure 4 —Ring architecture 


networks is the one sponsored by the ARPA, and just as 
naturally, these algorithms have been biased by the AR¬ 
PANET environment during that period. The fourth section 
describes these biases in the algorithm, and specifies a mod¬ 
ified algorithm that eliminates these biases and improves 
both their effectiveness and efficiency. 7 

In the past, the number and locations of the NAF’s have 
not been an important consideration for optimization. They 
are more constrained by users’ organizational structure than 
costs. As a consequence, algorithms developed to date are 
more of an academic interest than for real network optimi¬ 
zation. Hence, definite room exists for the improvement in 
these algorithms’ effectiveness and related efficiency. In 
particular, even though the costs of the local access and 
backbone networks depend on the number and locations of 
the NAF’s, these algorithms have not sufficiently considered 
these configurations in making decisions on the number and 
locations of the NAF’s. The fifth section develops an effec¬ 
tive and efficient algorithm for the determination of the 
number and locations of the NAF’s. 




Q TERMINAL CONTROL UNIT (TCU) 
I""] UNINTELLIGENT TERMINAL 

Figure 5—TCU-terminal configurations 


Using conventional approaches to design a distributed 
network as shown in Figures 2, 3, or 4, the number and 
locations of the NAF’s, the configuration of the local access 
network, and the configuration of the backbone network are 
each treated as a separate optimization problem. The opti¬ 
mal network cannot be realized by simply overlaying three 
optimized subnetworks. To avoid such a suboptimum de¬ 
sign, it is necessary to integrate consideration of the sub¬ 
networks in the final and overall optimization process. The 
fifth section, in conjunction with the determination of the 
number and locations of NAF’s, introduces an integrated 
approach that combines the three optimization problems 
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into one. This integrated approach, together with the algo¬ 
rithms given in earlier sections, has been implemented as a 
computer program. It is capable of designing in one single 
execution, a distributed network of the form shown in Fig¬ 
ures 2, 3, or 4. In addition to the integration of three optimi¬ 
zation problems into one process, the program which has 
been developed also integrates into the same process the 
different network shapes (tree, ring, mesh) and variations 
associated with differences in line circuit tariffs. 


LOCAL ACCESS NETWORK OPTIMIZATION—TREE¬ 
SHAPED MULTIPOINT CONFIGURATION 

The basic problem of tree shaped multi-point line design 
can be stated as: given a geographic distribution of termi¬ 
nals, locations of roots (network access facilities, host com¬ 
puters), line speeds or capacities and constraints (such as 
maximum number of terminals permitted per line, maximum 
traffic permitted per line as a function of the terminals on 
the line), design a least-cost network connecting the termi¬ 
nals to the roots. 

Many, mainly heuristic, techniques have appeared in the 
literature for the optimization of tree-shaped networks in 
data communication systems. 2 * 3 * 8-11 The techniques, though 
differing, are all constrained minimum spanning tree algo¬ 
rithms. They design minimum cost tree-shaped networks 
satisfying the constraints. Each of these algorithms starts 
with the terminals in separate components and subsequently 


joins pairs of components. They differ only in the order in 
which they consider joining components. If no constraints 
are imposed, they all yield a least-cost tree-structured net¬ 
work. Since the grouping of a given set of terminals into one 
component (i.e., placing them on the same multidrop line) 
restricts subsequent merging with other components be¬ 
cause of the constraints, these algorithms, in general, yield 
different solutions. For example, Prim’s algorithm 9 chooses 
a terminal whose distance to any terminal already in the tree 
is minimal as a basis for joining components to the tree. 
Kruskal’s algorithm 8 evaluates a least-cost line connecting 
a pair of components and joins the components by that line. 
(See Reference 3 for more details.) 

Based on the above principle, Chou and Kershenbaum 3 
have developed a unified algorithm which, with the proper 
choice of parameters built into the algorithm, allows one to 
emulate a specific algorithm and to take advantage of the 
precepts forwarded by many of the existing algorithms. The 
network designer, using the Chou-Kershenbaum algorithm, 
can take advantage of features offered by other algorithms 
or experiment with new algorithms by simply varying pa¬ 
rameters. In addition, while most techniques allow only one 
root for all the tree networks during the execution of the 
algorithm and, in the case of multiple roots or NAF’s, re¬ 
quire a preprocessing step of clustering terminals for each 
of the roots or NAF’s, the Chou-Kershenbaum algorithm 
allows multiple NAF sites to be considered during the exe¬ 
cution of their algorithm. A modified version of the algo¬ 
rithm is given below. 


CHOU-KERSHENBAUM’S UNIFIED ALGORITHM FOR TREE NETWORK OPTIMIZATION 


STEPO INITIALIZATION 
AM 1,2, . . . ,N t } 

BMNt + 1, . • • ,N t + N SAF } 

CA{i}VieAUB 

Define W { 

tij Wj dij , ieA 

jeAUB 

N l = 0 


(Set of all yet unconnected terminals, 7V t =Number of terminals) 

(Set of all concentrators, A NA F=Number of concentrators) 

(Initially, each component C; contains one node) 

(Weighting function associated with C t . e.g., min. connecting cost from C t to any 
concentrator) 

(Cost trade-off function of connecting C; and Cj, da =cost of connecting i and j) 
(Number of connected links=0) 


STEP 1 DEFINE MAX COST TRADE-OFF FUNCTION 

tiV-Max ta 

ieCiHA (/ must be a terminal) 

jeCj (j may be either) 

Ci#Cj (/ and j not in same set) 


STEP 2 ADD LINKS 

If Ci*UCj* does not violate constraints, 


Add (/*,/) 

Let Q**— Cj*UCj* 

n l <-n l +1 

Update Wi and t v . 
Otherwise, let ti*j*=°° 


(Adding a link) 

(Merging two components) 
(Increment N L ) 

(Wi may vary with configurations) 
((/*, j *) are not allowed) 


STEP 3 CHECK NUMBER OF CONNECTED LINKS 

If N L <N t , go to Step 1 (End of the algorithm if the number of links equals to the number of terminals) 

Otherwise, stop. 
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The algorithm does not restrict the definition of the ter¬ 
minal weight, Wi. However, Chou and Kershenbaum 3 sug¬ 
gests the following definition: 

Wi=a(bx di 0 + (1- b)x dm) 

Where, 

d i0 =C ost of connecting terminal i to the closest NAF 
d t 2 =Cost of connecting i to the second nearest neighbor 
a, bare parameters of the algorithm such that a> 0, 

The parameters, a and b, when varied allow the Chou-Ker- 
shenbaum algorithm to essentially perform operations sim¬ 
ilar to other algorithmic processes. For example, by setting 
a= 0 Kruskal’s algorithm is simulated; or by setting a = b=\ 
Esau-William’s algorithm results. 

Before, the publication of References 3 and 12 proved that 
most prior published heuristic algorithms achieve near-the- 
oretic-optimum results, particularly so of the Esau-Willi am 
algorithm. 2 Cain 13 then showed that Chou-Kershenbaum al¬ 
gorithm always outperforms Esau-William algorithm. Thus, 
the effectiveness of the algorithm is well recognized. How¬ 
ever, the efficiency of the algorithm, i.e., the computer 
memory size and running time required to execute the al¬ 
gorithm, has not been shown to be optimized. 

In a large network with many terminals, large memory 
requirement and long execution times are primarily the re¬ 
sult of determination and updating of t i} values. Technically, 
determination of minimum t tj implies the computation and 
storage of (t i} ) for all / and j. With 1,000 terminals, this means 
that almost 1,000,000 lines have to be considered for costing, 
at each stage of the algorithm. The memory and computation 
time requirements can be expressed as, for large N, 
K m N am and K t N at , where K m , a m , K t , a t are constants, and 
N is the number of terminals. With a straightforward imple¬ 
mentation, a m is 2, a t is 2 or 3. 14 The computation time 
could be horrendous and the algorithm becomes impractical 
to use even if N is in the low hundreds. 

In order to improve the efficiency computation time, we 
must lower the value of a m and a t . It can be observed that 
by proper preprocessing, the value of K t is increased with 
companion consequence of lowering the value of a t . With 
the following two schemes, it can be shown that we are able 
to reduce a m and a t close to “1”. For large practical net¬ 
works, a m and a t can actually be reduced to a value less 
than “1.” (The net result is that, memory storage size and 
computation time are increased by the schemes for small N 
due to the increase in K t , K m ; but are reduced for large N .) 

SCHEME A—ONLY TERMINALS WITHIN A 
NEIGHBORHOOD BEING CONSIDERED FOR 
CONNECTIONS 

Without significant degradation in the resulting solution 
the evaluation can be reduced to consideration of only a few 
tij, based on the premise that in the optimum design net¬ 
work, it is very unlikely that a terminal will be connected 
directly to a very distant terminal. 

Significant execution time savings can be obtained in de¬ 
termining the nearest neighbors if, instead of evaluating all 


inter-terminal costs, the area containing the terminals is 
partitioned into grids. Then, for each terminal, the nearest 
neighbors are found by considering only the terminals con¬ 
tained either within the same grid or in the next adjacent 
grid area. 

The evaluation and updating of can be speeded up by 
maintaining the neighbors as a list-data structure with as¬ 
sociated nearest neighbor trade-off parameters. This struc¬ 
tured list is referred to as a “heap.” Updating a “heap” 
structured array can be shown to be much more efficient 
than re-sorting an identically sized array. This is due in part 
because the least element is always located at the top of the 
heap. Maintaining the neighbors in a linked list has the 
advantage that after merging link (/) with its immediate 
neighbor link (j), by simply adjusting appropriate pointers, 
immediate neighbors of link ( j) can be assigned as link (i)’s 
immediate neighbor. Note, once link (j) has been merged 
with link (/), it is no longer necessary to consider the link 

( i, j)- 

This scheme has been implemented with FORTRAN in 
an IBM 370/158. Figure 6 illustrates the relationship between 
computation time and the number of different terminal lo¬ 
cations. As shown, the curve is practically linear. Super¬ 
imposed on the Figure is the predicted curve that would 
have occurred had this scheme not been employed. Let K t , 
for the curve representing the implementation with Scheme 
A be K t ', and for the curve representing the implementation 
without Scheme A, be K t ". We conservatively assume 
K t ’=50K", and, optimistically assume a t = 2 for the curve 
without Scheme A. (K t ' is found to be 0.75). 

With Scheme A, the computation time can be expressed 
as Co + Cj log N+K t 'N. For large A, the term Co + Cj log N 
is insignificant. 

SCHEME B—PREPROCESSING TERMINALS IN A 

SAME LOCALITY 

When terminals served by the same exchange are inter¬ 
connected often no mileage charges are levied. Thus they 
can form components and be interconnected based only on 
traffic constraints. By only increasing the number of termi¬ 
nals in each locality without increasing the number of lo¬ 
calities, the value of “iV” in K t N' M is hardly increased and 
the computation time is only increased slightly. For systems 
with more than a couple of hundred terminals, it is likely 
that many locations have multiple terminals. The multiplicity 
of terminals per location increases as the total number of 
terminals increases. Thus for systems with several hundred 
terminals or more, the computation time goes up much less 
than linear as the terminal size goes up. Figure 7 illustrates 
this relationship. 

LOCAL ACCESS NETWORK OPTIMIZATION—RING 

NETWORKS 

At the present, only a very small percentage of data and 
computer communication networks are ring-shaped. For 
those that are, the nodes on the ring network are concen- 


COMPUTATION TIME (Sec.) 
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trated within a building complex, or the number of nodes is 
small. Eyeball design of such networks has been adequate. 
And no algorithm has been published for the design of ring- 
shaped computer communication networks. To consider 
ring-shaped networks covering larger geographical areas, 
good algorithms are needed. Good algorithms do exist; how¬ 
ever, they were not inspired by data and computer com¬ 
munication networks, but are directly applicable to the lay¬ 
out of ring-shaped data communication networks. 

The topological layout problem for the ring-shaped data 
networks can be classified into two problems. One problem 
is to connect terminals or communication devices into a ring 
(loop) shaped network with minimum total transmission line 
cost, given the locations of the devices. This problem is 
equivalent to the traveling salesman problem of determining 
a minimum cost tour for a salesman visiting every city ex¬ 
actly once. It can be effectively solved by following steps: 
start from an arbitrary initial ring interconnecting the nodes; 
interchange one or more pairs of links while keeping the 
network in the ring-shape; if the exchange does not result 
in a net cost saving, restore the original network; otherwise, 
use the new network and repeat the interchange process 
until no appreciable improvement can be made. 5 

A second problem involves the layout of multiple ring- 
shaped networks. In this case, the number of terminal de¬ 
vices that can share a loop is often constrained by the ter¬ 
minal generated traffic and by the link capacities. This sec¬ 


ond problem is the constrained loop problem, and can be 
considered an extension of the traveling salesman problem. 
Instead of having one salesman visiting all the cities, there 
are now several salesmen visiting different cities. Each sales¬ 
man must start from the central office. The number of sa¬ 
lesmen and the number of cities to be visited by a salesman 
depends upon the sales load at the cities. The problem can 
be solved effectively by partitioning the terminals into sep¬ 
arate rings and then optimizing each of these rings as a 
separate traveling salesman problem. 

A very good algorithm for partitioning the terminals is the 
following one given by Clark and Wright. 6 Initially, consider 
every terminal as forming a separate ring connected to the 
central node (communication device or host computer) by 
two links. Merge the two rings which result in the largest 
cost saving without violating constraints. Each time two 
rings are merged, two of the four links connecting them to 
the central node are removed and one link connecting the 
two individual rings is added. This process continues until 
no more rings can be merged without violating the con¬ 
straints. This Clark-Wright algorithm gives good results for 
the terminal layout design problem even without the link 
exchange reoptimization of the individual rings. It is also 
interesting to note that their algorithm is quite similar to the 
one for tree network design. The Clark-Wright algorithm is 
presented below in a form similar to tree network design 
algorithms. 


MODIFIED CLARK-WRIGHT ALGORITHM FOR RING NETWORKS 

STEPO INITIALIZATION 
A={0,1,2,3, . . . ,N t } 

Ci = {i} V/eA 
ij = i 2 = i Vie A 

dh = di 2 ~ di V/eA 
ta = fe= MAX {d ik + d jh - d ikJh }V i,j,eA 

n l = o" 

STEP 1 DET. THE MAX COST TRADE-OFF FUNCTION 

ti*j*-Maxtjj i,jeA 

STEP 2 MERGING TWO LOOPS BY ADDING A NEW LINK 

If C,*U Cj* does not violate constraints, (e.g., no more than certain number of terminals per ring) 

Add ( i*,j*) 

Cj**—Ci* U Cj* 

A<-A-{/*} 

Det. j>, j 2 *, d h *, dj 2 * 

Update t u *, or tj* A for i+ j*eA 
N l = N l + 1 

Otherwise, t t *, = —<» 

> STEP 3 If N L <N t Go to Step 1 

Otherwise, add a link from 0, the NAF, to the free end point of CjVieA, 

STOP 


(Set of all nodes, N^Number of terminals) 

(Initially, each component C t contains one node) 

(/! and i 2 denote the two end points of the chain formed by 
C t ) 

(d k =the cost connecting k to NAF) 

(Det. the cost trade-off functions of merging C 4 , C } ) 

(Number of links=0) 
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BACKBONE NETWORK OPTIMIZATION—PACKET 

SWITCHING NETWORKS 

A backbone network can be point-to-point, tree-shaped, 
ring-switched, or packet switched. For the first three types 
techniques and algorithms given in earlier sections apply. 
This section discusses the optimization of packet switching 
networks. 

The topological design of distributed packet switching net¬ 
work was first discussed by Frank, Frisch, and Chou. 15 They 
described the application of branch exchange techniques in 
conjunction with branch deletion and addition techniques to 
the optimization of ARPANET. A summary of experience 
with branch exchange techniques for ARPANET optimiza¬ 
tion is given in Reference 16. Similar techniques are dis¬ 
cussed in Reference 17. Basically, these techniques seek 
repetitive cost or throughput improvement by adding, de¬ 
leting, or simultaneously adding and deleting links, 
(branches, lines) to a starting network until there are no 
modifications that result in an appreciable improvement. An 
evolution from the branch addition, deletion, and exchange 
is the cut-saturation algorithm 7 which improves both the 
effectiveness of the design and the efficiency of the com¬ 
puter running time. A cut is any set of lines whose removal 
will disconnect the network. A cut is saturated if the traffic 
load of every link in the cut is at capacity. There are, typi¬ 
cally, a large number of cuts in a network. When traffic load 
grows, one of the cuts approaches saturation. Then the only 
way the capacity of the network can be increased is by 
increasing the capacity of at least one link in the cut or by 
adding links across the cut. These ways form the basis for 
the cut-saturation algorithm. The algorithm attempts to sat¬ 
isfy throughput and response time requirements while keep¬ 
ing overall cost low. It has been successfully applied to the 
various stages of the ARPANET and the AUTODIN II, 
among others, and has been shown to be near optimum for 
problems under consideration. 18 

Another class of heuristic algorithms for designing packet 
switching networks are called concave branch elimination 
algorithms. 7,18,19 However, their limited applicability makes 
them less useful than the cut saturation approach. 

However, the cut saturation algorithm as given in Refer¬ 
ence 7 is somewhat biased by the ARPANET environment 
and is limited in its applicability to more general packet 
switching networks. Specifically, (1) it always starts with a 
good starting design and uses the optimization algorithm to 
improve upon it, (in a general environment, a “good” start¬ 
ing design is not always available.); (2) it pays little attention 
to the applicability of multiple capacities and parallel links, 
(with the proliferation of line tariffs, such considerations are 
a must.); (3) the criterion to determine the locations to add 
links across a cut is derived purely from the experience of 
ARPANET network structure, (therefore, it is likely not 
well suited to other environments). A unified algorithm for 
packet switching network optimization has been developed 
with those deficiencies eliminated. 20 


Step 1—Generating a starting network 

(a) Let TRM(/c)=Ath largest element in {TR(/,j')} where 
TR (i,j) is the traffic requirement from packet switch 
i to j. 

(b) Reorder first N t elements in TRM according to as¬ 
cending link costs. (N t is determined by a prespecified 
parameter A, such that TRM(1)/TRM(./V 1 )^A< 
TRM(1)/TRM(A 1 +1). Let the vector be TRCOST. 

(c) Let the end points of the line represented by TRCOST 
(1) be i and j. If the sum of the line capacity connected 
to k divided by 2TR(fc,0, k=i or j, does not exceed 
a preset limit, B, line (ij) is added. Otherwise, elim¬ 
inate this link from the list and a new link is picked 
from TRM and TRCOST is reordered. 

(d) Process (c) is repeated until all the nodes are con¬ 
nected. 

Parameters A and B impact on computer running time. 
For small network, they may be set to infinity. For large 
network, they should be set small. But if too small, a feasible 
network cannot be realized. 

Step 2—Define generalized link capacities 

Let (Si | i=l, . . . N s ) be the set of available link speeds 
(capacities). The set of generalized capacities (C m | C m <C m+1 , 
m— 0,1,2, . . .) are defined as: 

C 0 =0 

Ns 

C m = ^ niiSi m= 1,2,3, . . . 

i —1 

A link can assume a generalized link capacity C m . C m , in 
general, consists of parallel lines of different speeds or ca¬ 
pacities. Among them there are parallel lines with speed 
St, for i=l,2, . . . ,N S . The value of mi can be zero. 

Step 3—Routing 

The routing operation checks whether a network can sup¬ 
port the required throughput or can support more traffic 
than the original network. It is performed after each network 
modification and is used to generate a new optimal link flow. 
The routing operation is a necessary step regardless of which 
topological optimization technique is used. 21 

Step 4—Saturated cutset determination 

The cutset that has been newly saturated must be deter¬ 
mined after each operation. Link flows are calculated. The 
minimal cutset that has the highest utilization is the saturated 
cutset. 
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Step 5 — Add-only algorithm 

If the network structure under consideration cannot ac¬ 
commodate the required throughput, an add-only algorithm 
determines the “best” link to be added across the two com¬ 
ponents separated by the saturated cutsets. 

Let C„ k be the generalized capacity on link k. The ‘adding’ 
of a link is used in a generic sense to mean the upgrading of 
the value of capacity for link k from C„ fc to C nk+1 or higher. 
If link k did not exist before, this operation becomes the 
addition of a new link. Otherwise, this operation may up¬ 
grade the capacity of individual lines in link k or add parallel 
lines to link k. 

Let D k (C m ) be the line cost of link k if link k assumes a 
capacity value of C m . Link k must be either an existing or 
a potential new link across the saturated cut. Either of the 
following two criteria may be used: 

Criterion 1. Upgrade the link whose upgrading results in 
minimum additional costs, that is, upgrade 
link K if 

DK(Cn K +i) — D K (C nK ) 

—min{Dfc ( C nk +\ ) D k (C^)} 

k 

Criterion 2. Upgrade the link whose upgrading appears to 
result in minimum additional costs per bit, 
that is, upgrade link K if 
{{D K {C nK+ r)-D K (C nK ))/{C nK+ -C nK ) 

=min (( D k ( C nk+1 )-D k (C % ))/ (C %+1 - C „) 

For computer running time efficiency, only a subset of 
possible k' s should be considered. In particular, choose only 
those links whose end points have high unsatisfied traffic 
requirements for nodes across the saturated cut. 

Step 6 — Delete-only algorithm 

This algorithm may be applied if the throughput supported 
by the network structure under consideration is higher than 
the requirement. The Delete-Only operation begins with a 
highly connected topology and “deletes” one “link” at each 
iteration, continuously reducing cost and throughput. The 
deleting of link k is used in a general sense to mean the 
degrading of link k's generalized capacity from C„ fc to 
C njc _i or lower. If C^-i is zero, this operation eliminates 
link k. Otherwise, this operation reduces the capacities of 
individual lines in link k or reduces the number of parallel 
lines in link k. 

Either of the following two criteria may be used for “de¬ 
grading.” 

Criterion 1 . Degrade the link which has the most residue 
capacity, that is, degrade link K if 
C^-i-/«=max(C„ Jt - 1 -/ fc )>0 where f k is 

k 

traffic flow on link k. 


Criterion 2. Degrade the link whose degrading will result 
in the maximum costs per bit savings, that is, 
degrade link K if 

(DACn^-DACnn-^/i f K ~C nr _ t ) 
=max{(D fc (C„ k )-D k {C Kk - x ))/{ f k ~C nk -i)} 
for f k < C nk —x 

Step 7 —Exchange 

This operation combines Add-Only and Delete-Only op¬ 
erations and is used to improve the throughput or the cost. 
One link is introduced according to the Add-Only algorithm 
and one link is removed according to the Delete-Only al¬ 
gorithm. Perturbation is performed when the Add-Only al¬ 
gorithm cannot further improve appreciably, the throughput 
and the Delete-Only algorithm cannot further improve ap¬ 
preciably the cost. 

Figure 8 demonstrates the efficiency and effectiveness of 
the unified algorithm. Figure 8(b) is an AUTODIN II net¬ 
work design that is designed with several executions of 
ARPANET-based optimization algorithm, and with substan¬ 
tial amount of human interface, including the human config¬ 
uration of starting network design. Under the same through¬ 
put and response time requirements, Figure 8(a) is the design 
obtained by a single execution of the unified algorithm with¬ 
out input specifying a starting network. It was designed with 
28 sec of CPU time on an IBM 370/168 machine which 
results in a computer charge of $2.20. Yet this design realizes 
savings of over $200K (11 percent) annually. Figure 8(c) 
summarizes the comparison. 

INTEGRATED OPTIMIZATION AND NAF 

PLACEMENT 

Integrated optimization process 

The communications cost for a distributed system in gen¬ 
eral consists of the costs of NAF’s, local access network, 
and the backbone network. Minimizing the costs for one of 
the three elements is likely to increase the overall cost. It 
is important that to optimize the overall communication 
network costs, the three design problems should be treated 
as an integrated one. All conventional approaches, however, 
have treated them as separate sub-optimization or design 
problems resulting in potentially an overall non-optimized 
network. We present here an integrated approach. 

Since the number and locations impact directly on the 
costs of both local access and backbone networks, it is only 
logical to implement the integrated optimization process in 
conjunction with the optimal placement process of the 
NAF’s. Figure 9 is a simplified version illustrating the in¬ 
terrelation of the integrated process and the three optimi¬ 
zation processes. Basically, whenever a set of locations are 
considered as potential NAF sites, the integrated process 
interfaces the local access optimization and the backbone 



( ) NUMBER OF 56 Kb/s 
DATA TRUNKS 



a. Design wi th Unified Algorithm 



and Human Intervention 

Figure 8-Comparison of network design with and without the unified 
algorithm 
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Network Throughput 
Avg. End-to-End Delay 
Comm. Cost/Yr. 
Computation Time 

Computer Time Charge 
Human Interface 


Unified Algorithm 
1529.2kb/sec 
0.1233 sec 
$1,675,411 
28 sec 

(IBM 370/168) 

$ 2.20 

None except 
input preparation 


ARPANET Algorithm 
1635.434kb/sec 
0.1233 sec. 
$1,881,474 
Substantially More 

Substantially More 
Substantially More 


c. Comparison of Designs (a) and (b) 


network optimization process to determine the costs of the 
two subnetworks under this set of NAF’s. The network 
costs may either be determined by a complete optimization 
procedure or by heuristic estimation. The next section de¬ 
scribes the algorithm combining the integrated process and 
the NAF optimization process. 

Optimization of NAF placement 

The problem of determining the number and location of 
NAF’s is very similar to the classic warehouse (or plant) 
location problem. The warehouse location problem is to 
determine the number, location, and capacity of source sites 
which minimize the cost of satisfying shipping require¬ 
ments. 31 A “warehouse” corresponds to an “NAF,” and 
shipping requirements correspond to traffic requirements. 
Many techniques and algorithms for warehouse location 
problems have been applied by many authors to the NAF 
problem. 

A general NAF location algorithm has (heuristic) proce¬ 
dures for as many as four problems: determining potential 
sites, iteratively placing NAF’s at the potential sites, esti¬ 
mating the cost of local access networks, and estimating the 
cost of backbone network. 

If the number of sites on which NAF’s may be placed is 
small, every one is designated as a potential site. If not, 
only a subset of all the possible sites are designated as 
potential sites to reduce the computational complexity. It¬ 
erative procedures are then used to determine the number 
of NAF’s and the eventual NAF sites among the potential 
NAF sites. The optimal number and locations of NAF’s 
depend on the costs of the network because NAF’s interface 
local access networks and backbone networks. These net¬ 
work costs are sometimes estimated instead of being deter¬ 
mined exactly. 


Many algorithms have been developed and published, 
e.g., References 22 through 28. However, they all have only 
limited applicability. This is probably because there has been 
very little demand for using algorithms to place the NAF’s. 
There are very few network designs in which the NAF 
locations have been determined by any of the algorithms 
referenced above. 

For large communication based distributed data process¬ 
ing systems, more flexible and more effective algorithms are 
needed. In particular, there are the needs for better evalu¬ 
ation of both local access and backbone network costs, and 
the needs of more flexible algorithms in considering the 
alternative NAF site. Described below is an algorithm that 
has incorporated these needs. 29 

The integrated NAF location algorithm 

Step 1—Select a subset of permissible sites as potential 
NAF sites 

a. Check the size of locations permissible as NAF sites. 
Certain locations may be designated as mandatory 
NAF sites and certain others as permissible. If the sum 
of the two sets is reasonably small, they are all desig¬ 
nated as potential NAF (PNAF) sites and go to Step 
2 . 

b. Cluster Terminals—If the input specified PNAF set is 
large, the magnitude of the design problem may be 
reduced by initially clustering the terminals. Intui¬ 
tively, a good location for a NAF is the center of a 
cluster of terminals. Thus, by only considering the 
centers of a set of clusters, the magnitude of the initial 
problem may be significantly reduced. The number of 
initial clusters and the cluster size is either input or 
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estimated by the program. Then, using Kruskal’s al¬ 
gorithm, 8 the clustering is performed. 

c. Determine Center of Mass of the Clusters—The center- 
of-mass (COM’s) for the clusters are determined. Only 
those clusters that have permissible NAF sites but not 
mandatory NAF sites need be considered. For each 
such cluster, define the COM by: 

Nc 

V Coordinate of COM= £ UVi/H h 

i 

Nc 

H Coordinate of COM = X h 

i 

where ?*=traffic from Node i 

Vi,Hi=(V, H) coordinates of Node i 
A c =Number of nodes in cluster 

d. Refine the Center of Mass—The COM as defined 
above is a mathematically defined location and, hence, 
may not correspond to any physical n<5de location. 
Determine a permissible NAF site closest to each COM 
and assign it as a PNAF. 

Step 2—Design or estimate local access network 

This step designs the local access network. Initially, as¬ 
sume that all PNAF locations are NAF locations. Using 
algorithms in earlier sections, the local access network is 
designed. 

Step 3—Design or estimate backbone network 

Having designed the local access network, the traffic em¬ 
anating from each PNAF can be assessed. Four distinct 
types of backbone network designs are considered: (a) Point- 
to-point (b) Multipoint (c) Packet-switched (d) Ring. 

In (a), for each NAF, the closest CPU is connected. 

In (b), the NAF’s are connected to the CPU’s via multi¬ 
point trees using similar algorithms defined earlier. 

In (c), Packet-switching techniques are used for the back¬ 
bone network using algorithms defined earlier. 

Step 4 —Select the NAF sites 

a. Drop Algorithm—In this step, for each PNAF that is 
non-mandatory and has been selected as a NAF site in 
the current design step, the potential cost-savings to 
be obtained by dropping it is evaluated. This savings 
consists of: (a) savings in backbone network, (b) sav¬ 
ings in local access network. A PNAF that yields max¬ 
imum positive savings is permanently dropped from 
the network. If the savings is negative, this step is 
concluded. The savings (a) and (b) can be either eval¬ 
uated exactly or estimated using heuristics. 


• Heuristic Cost Estimate for Backbone 

If the PNAF considered for dropping is singly con¬ 
nected to another node, the backbone cost saving 
simply equals the line cost plus the physical cost of 
the NAF. If the PNAF is multiply-connected, con¬ 
sider all the nodes directly connected to this PNAF. 
Consider, in turn, each of these nodes for direct 
connection to the rest of the nodes. Whichever of 
these yields the least cost is considered the alterna¬ 
tive. The cost-saving equals the line cost of the new 
configuration minus the line cost of the previous 
configuration plus the NAF cost. 

• Heuristic Cost Estimate for Local Access 
Consider all the lines that were connected to this 
PNAF. For each of these lines determine the closest 
NAF (other than the NAF under consideration) and 
check if this NAF can accommodate one additional 
input port. If not, dropping the NAF is infeasible. If 
it can, consider this the alternative local access con¬ 
figuration. As before, the cost savings equals the 
new line cost minus the previous line cost. 

After each K (“K” is an input parameter) iterations, 
the local access and backbone networks are reopti- * 
mized. 

b. Add Algorithm—This step considers the cost savings 
to be obtained by adding an NAF site. For this purpose, 
PNAF’s that are not selected for the current design 
and that are farthest from the current set of PNAF’s 
are considered for potential addition. The cost savings 
are heuristically estimated. The PNAF whose addition 
results in the greatest positive saving is added to the 
network. If the saving is negative, conclude this step. 
If the number of iterations equals K, the local access 
and backbone networks are redesigned. 

c. Split Algorithm—This step considers the cost saving 
that results if a PNAF having a large number of ter¬ 
minals is replaced by two PNAF’s. Consider “KS” 
(input parameter) PNAF’s that have the most number 
of terminals. For each of these, heuristically estimate 
the cost saving by replacing it with two NAF’s. If the 
maximum cost saving is positive, the split is made 
permanent; if not, conclude this step. After K itera¬ 
tions, the local access and backbone networks are 
redesigned. 

d. Exchange Algorithm—This step examines the cost sav¬ 
ing obtainable by exchanging NAF locations. It con¬ 
siders unselected PNAF locations that are farthest 
from the current set of selected PNAF’s; each of these 
locations is examined for the effect of exchanging it 
with a selected PNAF closest to it. As before, an 
exchange which yields the most positive heuristic cost 
saving estimate is made permanent. After each K it¬ 
eration, the local access and backbone networks are 
redesigned. 

e. Merging Algorithm—This step examines the cost sav¬ 
ing obtainable by merging a PNAF which has a small 
number of terminals associated with it, with a neigh- 
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NO. OF SWITCHES 

1 

1 

1 

NO. OF TERMINALS 

50 

50 

50 

ANNUAL TOTAL COST 

$110,248 

$117,048 

$124,908 

NO. OF SWITCHES 

1 

1 

1 

NO. OF TERMINALS 

100 

100 

100 

ANNUAL TOTAL COST 

$159,120 

$175,536 

$187,344 

NO. OF SWITCHES 

1 

1 

1 

NO. OF TERMINALS 

200 

200 

200 

ANNUAL TOTAL COST 

$247,860 

$276,876 

$277,956 


(b) Cost Comparison Table 



COST BASIS 

CONCENTRATOR (NAF) 

$200/MONTH 

LINE CHARGES/MI. 

$0.50/MILE 

TERMINATION CHARGE 

$$40/TERMINATION 


(c) Charges Considered in Determining Communication Cost 


Figure 10—Cost comparison information 
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Figure 11—Integrated algorithm network design (200 terminals) 


boring PNAF which can accommodate these additional 
terminals. 

In any one design run, all substeps need not be 
executed. For a given design problem one of these 
steps may be more useful in achieving low cost net¬ 
works than others. The designer can experiment with 
the order and number of these heuristics that need to 
be called to yield a near-optimum design. 

Step 5—Refinement and reoptimization 

If any network adjustments have been made since last 
redesigning the network, reoptimize both the local access 
and backbone networks. 

Figure 10a compares the costs between the designs ob¬ 
tained from the integrated and non-integrated approaches. 
Cost information for the non-integrated approach was de¬ 
rived from data presented in the recent McGregor-Shen 


paper. 28 Only local access, backbone line cost, and concen¬ 
trator costs were considered in doing the comparative eval¬ 
uation, since these costs are used primarily in our optimi¬ 
zation process. Fixed termination charges of $40/termination 
were subtracted from the McGregor-Shen data in order to 
provide a basis for comparison. 

Locations of terminals used in the data presented in Figure 
10 were obtained from a uniformly random distribution of 
sites taken over a 2000 by 3000 mile rectangle. This distri¬ 
bution of terminals was provided again in order to have a 
basis for comparison between our integrated algorithm and 
the algorithm presented in the McGregor-Shen paper. 28 

The specific cost comparisons of the two approaches for 
three network designs are given in the table in Figure 10b. 
Figure 10c gives the cost information on lines and devices. 
Figure 11 is the plot of one of the network designs (200 
terminals design). In the above comparisons it should be 
noted that the reason for using McGregor-Shen’s data is that 
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the authors felt that the McGregor-Shen COM algorithm 
was demonstrated to be the best heuristic algorithm pub¬ 
lished to date. Based on our comparative analysis we feel 
that our integrated algorithm approach improves upon the 
McGregor-Shen algorithm and uniformly outperforms all 
other non-integrated approaches published to date. 


CONCLUSION 

With the fast growth in the communication-based distributed 
data processing systems, the emphasis on the objectives of 
the optimization has shifted. We have presented a set of 
integrated network optimization algorithms that are best 
suited for this environment. Some of the algorithms are 
newly developed; some are the modifications of the existing 
ones. 

There are needs for reducing, substantially, the compu¬ 
tation time for local access networks with a large number of 
terminals. We have developed two implementation schemes, 
heuristic in nature, to satisfy these needs. The computation 
time, with these schemes, grows practically linearly as the 
number of terminal locations grows; and, in practical net¬ 
works with more than two or three hundred terminals, the 
computation time actually grows less than linearly. 

There are some needs for topological optimization algo¬ 
rithms for ring networks, which have not been available in 
the open literature in the data communication field. We 
presented such algorithms, developed by modifications of 
works in the graph theory area. 

There are needs for improvements on the widely used 
packet switching network optimization algorithms that were 
developed with ARPA support. The improvements are to 
eliminate its bias from the ARPA environment and to im¬ 
prove the design results and computation times. We pre¬ 
sented a generalized algorithm to achieve these improve¬ 
ments. 

There are needs for more effective algorithm than cur¬ 
rently available on the determination of the number and 
locations of network access facilities. To realize these needs, 
we presented an algorithm that adopts many good features 
of the existing algorithms and is augmented with new ones. 

Finally, there are the needs for integrating the different 
optimization processes into one unified process, which 
would turn out better design results. We presented such a 
process in conjunction with the NAF algorithms. In the 
program that we have developed with the implementation of 
these algorithms we were also able to integrate into the same 
process the different network structures (tree, ring, mesh) 
and variations associated with differences in line tariffs. 
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An extensible distributed 
data base system* 

by MAMORU MAEKAWA and SATORU ISHII 

Toshiba Research and Development Center 
Kawasaki, Japan 


INTRODUCTION 

Extensibility is one of many claims of improved performance 
being made for distributed processing systems. Extensibility 
means system properties, such as “incremental growth and 
configuration flexibility,” “highly adaptable to changes in 
workload,” “incremental replacement and/or upgrading of 
components (both hardware and software)” and “easily ex¬ 
panded in both capacity and function.” 2 Although the need 
for extensibility is strongly felt, research results for it appear 
to be still far from satisfactory. 3,4 

A distributed data base system is one example of distrib¬ 
uted processing. Although its usefulness is well recognized, 
the research in this area is also still very much in its infancy. 
MINIMET, a transaction-oriented homogeneous distributed 
data base system, has contributed towards the understand¬ 
ing of the distributed data base system. 6,9 The MINIMET 
research put more emphasis on the lower level of system 
control so that any data model can be built on it. 

The data base system described in this paper is a relational 
data base, one of whose characteristics, among others, is 
capacity extensibility. 

Extensibility, in this paper, means the following three 
things: 

(a) Physical extensibility 

(b) Logical extensibility and data independency 

(c) Minimal performance degradation in data base 
growth. 

Physical extensibility means that the system can physi¬ 
cally be extended to meet a workload increase. Logical 
extensibility and data independency mean that the system 
can logically be extended and that application programs are 
not affected by system extension. The third characteristic, 
minimal performance degradation, means that degradation 
in performance, response times in particular, in the growth 
of data base is kept minimal. Because of these characteris- 


* This work has been supported by the Agency of Industrial Science and 
Technology, Ministry of International Trade and Industry, Japan, under the 
project Pattern Information Processing System. 


tics, the system will be called an Extensible Distributed Data 
base System (EDDS). 

For physical extensibility, it is apparent that the less hard¬ 
ware for interconnections is provided the more extensible 
the system is. Obviously, a system consisting of independent 
subsystems is then most extensible. However, this architec¬ 
ture provides no advantages in coordinating various kinds 
of information. Distributed systems must consist of auton¬ 
omous, but not totally independent, cooperating subsys¬ 
tems. The next most extensible architecture would be a 
common bus system, as shown in Figure 1, in that a number 
of subsystems are connected by an intersubsystem bus. Any 
more complex architecture would impose a greater difficulty 
in expanding the system. Particularly, a shared memory 
causes a great difficulty. 

In the above common bus architecture, the bus capacity 
is most important in determining system performance. As 
the workload of the system and the number of subsystems 
increase, the bus capacity must also be increased. One 
method to increase bus capacity is to add additional buses. 
Another is to replace the bus with a faster one. The first 
method is architecturally more complex than the second, 
but it has the advantage that no replacements are necessary 
to increase the capacity. The second method requires bus 
replacement, but it is architecturally simpler. Both ap¬ 
proaches are possible. Further details will not be gone into, 
it will simply be assumed that a common bus architecture 
is the basis of the present system. 

Another important aspect of extensibility is performance 
degradation in data base growth. There are two major areas 
which cause performance degradation; input/output chan¬ 
nels and the system itself. In order to avoid performance 
degradation due to the limitation in input/output channel 
capacity, it is assumed that the system is connected by n 
input/output channels to the user systems as shown in Figure 
2. This is one of the reasons that a centralized control should 
be avoided. If there is a centralized control, then any and 
all data and control information must go through it and the 
central control may become a major bottleneck. 

Like the intersubsystem bus, it is also necessary to try to 
minimize input/output channel replacement cost. For this 
purpose, additions rather than replacements are desirable 
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to user 
systems 


subsystem 



intersubsystem 

bus 


Figure 1—Common bus architecture 


when more capacity is needed. Both approaches are feasible 
and the architecture shown in Figure 2 is assumed to be the 
basis of discussions. 

Performance degradation in the system itself is mainly 
caused by speed limitation of the intersubsystem bus, which 
is necessary to transfer data and control information be¬ 
tween subsystems to assure proper system operation. The 
bus capacity is thus an important system parameter. Per¬ 
formance degradation is analyzed in a later section. 

As indicated earlier, the system described in this paper is 
a relational data base. The query language for the system is 
relational-calculus oriented and has the following three basic 
statements. 

Retrieval 
Get R | condition 

Those tuples that meet the “condition” are retrieved 
and placed into relation or view “R”. The condition is 
any relational calculus involving relations and views, 
which are specified only by name. 

Relation creation 
Create R | condition 

Relation “R” is created under the “condition”. 

Relation deletion 
Purge R 

Relation “R” is deleted. 

Logical extensibility and data independency are now de¬ 
scribed in more detail. 


LOGICAL EXTENSIBILITY AND DATA 

INDEPENDENCY 

Logical extensibility and data independency mean that 
neither internal nor external logic is affected by system 
extension. The approach described in this paper, whereby 
to obtain this property, is one of the extremes of modular 
approach. In a general modular approach, a number of mod¬ 
ules are prepared and put together to form a particular sys¬ 
tem. Extensibility is obtained simply by adding more mod¬ 
ules. Depending on the size of the modules, the cost of 
interconnecting them and flexibility of resulting systems 
vary. The present system requires the following: 


of only one subsystem. This implies that the subsys¬ 
tem must be able to perform all the functions required 
for the data base processing. 

(2) No additional programming work, either internal or 
external, should be required in order to add additional 
subsystems. This implies that every subsystem must 
be logically identical. 

The EDDS tries to obtain internal logical extensibility in the 
following way. 

(a) Any internal logic is exactly copied and placed in each 
subsystem. Therefore, no program moves are re¬ 
quired. Note that although user programs are allowed 
to be cataloged and executed at a user request, they 
are here considered to be data. 

(b) Data are separated into two categories: relations and 
internal data. Relations are made up of information 
directly created and accessed by users. Internal data 
are the information necessary for assuring proper sys¬ 
tem operation. These data are created and maintained 
by the system. Relations keep spreading over the sub¬ 
systems and only one copy of each is maintained at 
any time. A single relation can be spread over several 
subsystems. On the other hand internal data are du¬ 
plicated and placed in each subsystem. Some exam¬ 
ples of internal data are relation sizes, locations and 
structural characteristics. This information is distrib¬ 
uted among subsystems under a loose control. Here, 
loose control means that the control of mutual exclu¬ 
sions is neglected and, thus, there might be some 
discrepancy at some time between multiple copies of 
internal data distributed among subsystems. The cor¬ 
rectness of any decisions based on this information is, 
as a result, probabilistic. 

The above discussions focus on the internal logic of the data 
base system. The external logic’s logical extensibility is also 
very important, which is commonly known as “data inde¬ 
pendency.” Particularly, we are here concerned with the 
number of subsystems. Data independency requires that 
accesses to data base be made independently of the internal 
structure, including the number of subsystems. In order to 
obtain this data independency, the following basic approach 
is used. 

(a) Each request from a user system is broadcast to every 


n ’ 

channels 
to user 
systems 


subsystem 1[~ 


[subsystem 21 - 


subsystem 


3- 1 


intersubsystem 

bus 


(1) The system must be able to work even when it consists 


Figure 2—System architecture 
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subsystem. Note that, at this point, it is not yet known 
where the necessary relations reside and which sub¬ 
system is going to take care of the request. The re¬ 
quest itself does not contain any such information. 
The request is written in the query language men¬ 
tioned before. 

(b) Let every subsystem start to try the processing of the 
request, but make all the subsystems except the op¬ 
timal one drop out during the course of processing. 
Eventually, the request is taken care of by the optimal 
subsystem. The results are returned to the user system 
directly from it. 

A key characteristic of this approach is expressed by the 
term “eventually” in the above sentence. No central control 
exists and no one tells any subsystem when to quit. Each 
subsystem is autonomous. This approach can be realized by 
the following algorithm. 


Algorithm 

1. A user system issues a request. The request is accom¬ 
panied with the time when it was created. This created 
time, together with the user system identification, will 
be used as a unique identification of the request. 

2. The request is broadcast to every subsystem in the 
data base system. 

3. The request is placed in the waiting queue of each 
subsystem. 

4. When the turn of the request comes, its subsystem 
tests whether the processing of the request has already 
been started by another subsystem. If so, the subsys¬ 
tem simply discards the request; otherwise it analyzes 
the request and, using some criterion, it determines 
whether it should accept the request. If it has decided 
not to do so, then the request is simply discarded. 
Otherwise, the processing is started. The above crite¬ 
rion will be discussed later. 

5. A subsystem now starts processing the request and 
sends a signal to other subsystems to notify it. If the 
request involves a creation of a relation, an OPEN to 
a dummy relation is made prior to any operations nec¬ 
essary for the request. 

6. It is assumed that any relation must be OPENed before 
any accesses are allowed. When an access to a relation 
is made with an OPEN procedure, a check is made to 
determine whether an access to it has been made for 
the same request from another subsystem. If so, the 
subsystem terminates the processing of this request 
and simply discards it. Otherwise, it continues the 
processing. This checking mechanism will also be dis¬ 
cussed later. 

7. When the results are produced, they are simply re¬ 
turned directly from the subsystem which accepted the 
request to the user system that issued the request and 
no other subsystems and/or user systems will be in¬ 
volved. 


Next, the acceptance criterion mentioned in (4), notifying 
mechanism in (5), checking mechanism in (6) and an OPEN 
to a dummy relation in (4) are discussed. 

Acceptance criterion 

Each subsystem determines whether it accepts a request 
or not according to some acceptance criterion. 

This acceptance criterion should be aimed at obtaining a 
balanced load assignment with a minimal overhead. It must 
assure that at least one subsystem accepts a request. A 
simplest algorithm is to let each subsystem accept every 
request and attempt to try its processing. This algorithm is 
simple and uses no information about the request. We call 
this algorithm “all selection algorithm” for later references 
and assume that the queue discipline is first-come, first- 
served. 

The algorithm can be improved by utilizing information 
about the request. The following information is useful: 

(a) Which relations are to be referenced. 

(b) How often they are to be referenced. 

(c) Which of them are in the subsystem concerned. 

Suppose that request R t references relations D n , D i2 , , 

D im with frequency F n , F i2 , . . . , F im , each. Then, the total 
number of references is 

m 

1 F„. 

}=1 

Assume that a fraction W tj of relation D u resides in the 
subsystem concerned. Then, the probability that an access 
falls into the subsystem concerned is 


A natural acceptance criterion is then: 

accept if W^a 

for some threshold value a. If we take a=0, it becomes the 
all selection algorithm. If we take a= 1, many requests would 
be missed and there would be a very high probability that 
a request would not be served by any subsystem at all. The 
maximum value that insures the acceptance of a request by 
at least one subsystem is 

n 

where n is the number of subsystems. We call the above 
improved criterion “sufficient selection algorithm” for later 
references. 

Notifying mechanism 

The request acceptance notification by a subsystem is not 
essential for proper system operation, but is employed to 
reduce overhead. The mechanism is a very simple one. Each 
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request is labeled with a unique identification, as mentioned 
before. Whenever a subsystem starts processing, it broad¬ 
casts a signal to notify every other subsystem of this. No 
control is placed on the work of this mechanism. Assurance 
of the proper operation of this mechanism requires a system- 
wide or intersubsystem mutual exclusion of signal setting, 
resetting and checking. Without a shared memory, this im¬ 
plementation of intersubsystem mutual exclusions is a rather 
time consuming process. Rather than trying to make the 
mechanism complete, instead it is made probabilistic. In 
other words, no intersubsystem mutual exclusions are em¬ 
ployed, although intrasubsystem mutual exclusions are guar¬ 
anteed. This simplification causes cases where, due to the 
delay in signal transferring, a subsystem checks the signal 
before it arrives. Therefore, a subsystem may proceed in 
spite of the fact that another subsystem is already process¬ 
ing. This causes some loss in processing power due to du¬ 
plicated work, but the mechanism itself does not need a 
difficult control and the architecture can remain simple. 
Note that a logic can be built so that there are no opposite 
cases; namely the case that the signal is set while no sub¬ 
systems actually have started. This guarantees a necessary 
request acceptance condition wherein a request is always 
accepted by at least one subsystem. 


Checking algorithm 

The checking algorithm is the last gate that a request must 
go through. If a request is passed through this check, then 
it will be processed through the completion. The checking 
algorithm is implemented in the OPEN procedure for a re¬ 
lation. Each OPEN procedure has a list of processed re¬ 
quests augmented with their subsystem’s names. Whenever 
an OPEN is requested, it checks whether or not another 
OPEN for the same request from a different subsystem has 
already been made. If so, the OPEN procedure returns con¬ 
trol indicating this and the subsystem then simply discards 
the whole request. It is assumed that any relation is always 
OPENed by its owner subsystem. This assures the mutual 
exclusion of OPEN procedure and makes the checking 
mechanism much easier. Note that it is assumed a relation 
is owned by a single subsystem although it may be spread 
over several subsystems. 


OPEN to a dummy relation 

An OPEN to a dummy relation is issued to assure proper 
relation creation. For relation creations, the following prob¬ 
lems must be considered: 

(a) How to control a relation creation so that one and 
only one is properly created. 

(b) How to distribute relations in a balanced way. 

The first problem is specific to the EDDS, in which a number 
of subsystems essentially try to do the same operation in 
regard to the same relation, including a relation creation, at 


the same time. Thus, when several subsystems try to create 
a relation, operations must be controlled so that one and 
only one subsystem will do the job. For this control, an 
access is defined and attached to a dummy relation to each 
request that involves a relation creation. It is assumed that 
this access to the dummy relation is made at the top of 
processing the request. In this way, through using the above 
checking algorithm, it will be assured that one and only one 
subsystem completes the operation and that no extra oper¬ 
ations are made. A small problem is how to determine the 
name of the dummy relation. The naming mechanism should 
have the characteristics: 

(a) Dummy relations are distributed to each subsystem 
so that the system can operate irrespective of the 
number of subsystems. 

(b) Accesses to the dummy relations are evenly distrib¬ 
uted among subsystems. 

In order to accomplish these characteristics, the following 
method is adopted. 

(a) A dummy relation is assigned to each subsystem, 
whose names are subsystem numbers 1 through n . 

(b) Each request is accompanied with its created time, t. 
The name of the dummy relation for this request is 
computed as (modulo n of t)+l- 

In this way, it should be obvious that the required charac¬ 
teristics are achieved, although the total number of subsys¬ 
tems, n, must be known to each subsystem. 

The second problem, how to distribute relations in a bal¬ 
anced way, is another important problem. A simple and 
perhaps natural way to obtain this is to let the first available 
subsystem accept the request and create the required rela¬ 
tion. This method sets up a relation to be assigned to a most 
likely lightest-loaded subsystem. Another simple and natural 
way is to assign the relation to the subsystem that has the 
largest amount of empty space. Control of this method is 
also easy. However, the former load-oriented approach is 
taken, since it is easier when it is combined with data search 
operations. 


CORRECTNESS AND OPTIMALITY 

This section proves the correctness of the algorithm pre¬ 
sented in the previous section and shows that the algorithm 
is statistically optimal for system balancing or load sharing. 


Correctness 

Algorithm correctness will be proved by showing: 

(a) At least one subsystem accepts a request, 

(b) No more than one subsystem accepts a request, 

(c) No deadlocks occur 

(d) System is alive 
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Methodology 

The methodology used here is a slight extension of Boch- 
mann and Gecsei’s method. 1 It is extended by adding remote 
read and write operations, denoted by READ (a:=v) and 
WRITE (v:=a). A READ (a:=v) specifies an access to a 
remote variable v and assigns the value of v to local variable 
a. Some arbitrary time delay exists between the initiation of 
the action and an actual reference to variable v. Control, 
however, stays at the READ command until the action is 
completed. A WRITE (v: =a) causes a remote variable v to 
be changed to the value of local variable a. A WRITE action 
does not wait for the completion of the action and some 


arbitrary delay exists between initiation and an actual 
change in v. 


Model 

For simplicity, it is assumed that the acceptance criterion 
is the all selection policy and limits the number of subsys¬ 
tems to two. Also, the number of relations used by a request 
is limited to two. As will be seen, an extension to a general 
case is very straight forward. A description of each subsys¬ 
tem can be given, referring to the algorithm described in the 
previous section. 


(a) State transition diagram for subsystem k. 
(see Figure 3) 

(b) Variables 


Variable 


( Meaning 


Cj[l: n]: integer for each relation i 

r r .,1 h if the ith relation for the jth request is OPENed by the 

1 J 1 hth subsystem. 



^ null otherwise 




f 1 if the jth request is already being processed by some 

Sfctl:/*]: (0,1) for each 

subsystem k 

S k [ j]= J subsystem 




0 otherwise 


Actions 




Transition 

Enabling predicate 

Action 

Meaning 

Arrival 

at least one request is 

* 

take the next request, say 


waiting 


jth, from the waiting queue 

Notify 

SJ j]=0 

S k [j]:= 1; WRITE 
(Si[f |: =1) for 
i— 1 , 2 ,. . . i+k\ 

notify the start of processing 

Processed 

s*[y]=i 

; 

discard the request 

OPEN 

C«[ j']=null 

C h [j]: =k; 

the first OPEN of the jth 

OK 

ACfc[m]=null 
for any m^j 


request is for the hth relation 

OPEN 

C h [ j] A null 


another subsystem has 

Unsuccessful 



already OPENed relation h 

OPEN 

Cl /|= null 

Cl j]: -k; 

the first OPEN for the ith 

Good 

ACi[m]=null 
for any m+j 


relation 

OPEN 

Ci[j]* null 

5 

another subsystem has 

Failure 



already OPENed relation i 

Return 

true 

Cil j] - =null; 

after the completion restart a 



Cj[y]:=null; 

new cycle 


Verification 

The enabling predicates for OPEN OK and OPEN Good 
involve more than a single request. Because of this, dead¬ 
locks can occur between conflicting requests. Since these 
deadlocks can be avoided by a usual technique, it is assumed 
they will not be a problem. Then, it is only necessary to 
analyze the case for the single same request. 

Let us denote by (t x ,t 2 ) the global system state, where t x 
and t 2 are two subsystem states. Possible transitions of the 
global system can be drawn as shown in Figure 4. Note that. 


since the subsystems are logically identical, the order of 
executions of OPEN’s for the relations for the same request 
is the same in each subsystem. From Figure 4, we can prove 
the four properties of the system. 

(a) At least one subsystem accepts a request. 

This is verified because once the system makes a 
transition “Arrival,” it always comes to a state in¬ 
volving a subsystem state (5). 

(b) No more than one subsystem accepts a request. 

This is verified because no (5,5) states are allowed 
and because once a state (5,x) is reached, then state 



818 


National Computer Conference, 1978 



Figure 3^-State transition diagram 



(y,5) can never be reached for the same request, and 
vice versa. 

(c) No deadlocks occur 

Since no states are blocked, no deadlocks occur. 

(d) System is alive 

This can be shown because, after the system reaches 
state (5,x) or (y,5) it always returns to the original 
state (1,1). 

The above verification is limited to a special case, but an 
extension to a general case should be obvious and straight¬ 
forward. 

Optimality 

It can be seen that the algorithm described in the previous 
section contains so called automatic load sharing. Several 
methods for automatic load sharing have been proposed. 
LeLann’s two methods, 7 for instance, are: 

(a) Diffusion technique 

(b) Circulating vector technique. 

The above two methods and many others exchange infor¬ 
mation about the load of each subsystem. Based on this 
information, each subsystem determines whether it accepts 
a new request. The present method, under the all selection 
discipline, does not need any information exchange and can 
be proved to be optimal for load sharing in a statistical 
sense. The present scheme is free from any consistency 
problems caused by communication delays, because no in¬ 
formation exchanges are required. Note that any scheme 
that exchanges information may have consistency problems, 
due to communication delays. 

Model and proof 

The present method, under the all selection discipline, is 
statistically equivalent to a single queue with multiple serv¬ 
ers, because each request is placed in every queue. Any 
scheme that exchanges information corresponds to a system 
of separate-queues with some jockeying. Whatever jockey¬ 
ing is used, they cannot be better than a single queue as far 
as load sharing is concerned. This can be proved very easily. 
Some simple analysis for jockeying for a variety of strategies 
is made by Koenigsburg 5 and a detailed analysis of two 
queues is made by Maekawa. 8 

PERFORMANCE ANALYSIS 

For performance analysis, the data base is simply consid¬ 
ered to be a collection of pairs (attribute, value). They are 
put together to form records, domains and relations. In this 
section, the only concern is with their statistical character¬ 
istics. There is no concern over whether they are actually 
relations, domains or records. 

The present data base system consists of a number of 
subsystems, as shown in Figure 2. Each subsystem has a 
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Figure 5—Subsystem 


processor, main memory and secondary memory, as shown 
in Figure 5. In the main memory, there are two kinds of 
buffers; one is for data handled directly by its own processor 
and the other for accesses from other subsystems. The for¬ 
mer buffer contains a working set of data. It is assumed that 
both main memory and secondary memory are divided into 
pages of equal size and that a paging scheme is employed. 
A model of the whole system is shown in Figure 6. 

(1) There are m users and n subsystems connected by n 
input/output channels 

(2) Each user issues a request with rate k. 

(3) A request is first examined by its subsystem. It is 
discarded unless the subsystem decides to accept it. 
It is assumed that the initial examination time is gov¬ 
erned by random variable A and the probability of 
acceptance is denoted by a. Probability a is dependent 
on the system state. 



(4) The processing of a request is a cycle of an access to 
data, usually a tuple of a relation, and its processing. 
When data are not in the main memory, they are 
brought into the main memory from a secondary mem¬ 
ory. Time to access data in the secondary memory is 
assumed governed by random variable T. When data 
are obtained in the main memory, processing the data 
is started. Its service time is assumed governed by 
random variable S. On the completion of the service, 
request processing completes with probability q and 
repeats with probability (1 -q). Thus, the number of 
cycles per request is geometrically distributed with 
mean 1 /q. 

(5) The missing page rate is denoted by p. When a page 
is not already in a main memory, the required page is 
brought in from its own subsystem’s secondary mem¬ 
ory or from another subsystem. It is assumed that the 
probability that a page is brought in from another 
subsystem is (1 —r). If a page is in its own subsystem’s 
secondary memory, it takes only time T to bring the 
page in. If it is not, however, a request for data ac¬ 
quisition and the obtained data must go through the 
intersubsystem bus, whose times are governed by ran¬ 
dom variables U and V, respectively. 

The major performance problems in this system are 

(a) How much performance degradation is imposed in 
each subsystem due to this architecture. 

(b) How good is load balancing. 

(c) How much overhead due to this architecture is im¬ 
posed. 

The performance degradation is, to some extent, controlled 
by the acceptance criterion. The performance degradation 
of the system, excluding I/O channels, is mainly due to the 
following overhead. 

(a) An unsuccessful attempt for access to a relation per 
subsystem for each request may be necessary. 

(b) When a page is in another subsystem’s secondary 
memory, it must be brought in through the intersub¬ 
system bus. 

Overhead (a) is negligibly small, compared to the total proc¬ 
essing necessary for a request. Thus, overhead (b) is the 
major cause for performance degradation. Instead of ana¬ 
lyzing the whole system shown in Figure 6, the performance 
of each subsystem may be approximated by the model 
shown in Figure 7. The rationale behind this approximation 
is obvious. The number of subsystems affects probability r 


—nri-^6^ 


T -H_sJ 
1-r *- u A - - J- v J 


i-q 


]—<i> 


Figure 7—Approximation model 
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as well as transfer times U and V. As a simple comparison, 
make a mean-value analysis. 

(A) Single-subsystem case 

In this case, r=l. Then, the mean service time for a 
request is 
£[service timeL 

=£[cycle time]E[number of cycles]+£[ A] 

= (pE[T]+E[S])/q+E[A\ (2) 

(B) Multi-subsystem case 
/^service time] m 

=jE[cycle time]£[number of cycles]+E[A] 

= {pE[T-]+p(l-r)(E[U]+E[V]) 

+E[S]}/q+E[A] (3) 

Since E[A ] is small, compared to the total processing 
time, ignore E[A\ and then take the ratio of the two for 
comparison purposes. 

£ [service time] m 
Etservice time]i 

_ p+E[S]/E[T] _ 

p+p(l-r)(E[U]+E[U]+E[V])/E[T]+E[S]/E[T] 


4 subsystems, should be able to work with 2 to 6 percent 
performance degradation. When there is a fairly fast sec¬ 
ondary memory and £[S]/£[J]=l/20 is justified, then, for 
h= 0.7, or c=l mega-byte/sec, performance degradation is 
only 10 percent with 30 subsystems. At any rate, it is ob¬ 
served that the smaller the missing page rate, and the faster 
the secondary memory, the less performance degradation is 
incurred and the larger a system can be constructed. Thus, 
a faster buffer memory between main and secondary mem¬ 
ories would greatly improve system performance. A mag¬ 
netic bubble or CCD memory would be a great help here. 

Next, the acceptance criterion effect is analyzed. The two 
policies discussed before are compared. 

(a) All selection policy 

(b) Sufficient selection policy 

Again, only a mean-value analysis is made. The assumptions 
are as follows: 

(1) The subsystem reference frequency is described by a 
truncated geometric distribution with parameter (1 — w). 
Namely, a request Ri refers a subsystem with 
probability 


Table I shows the values of d for various values of 

h=(l-r)(E[U]+E[V])/E[T]. (4) 

It is assumed that E[U], is one-tenth of E[V], page size is 
2 kilo-byte, E[T] is 30 ms and (1-r) is 0.3. Then h and c are 
related by 

h=(\-r)(E[U]+E[V])/E[T] 

=0.022 /c. 

Under normal operation conditions, the missing page rate, 
p, should be less than 1/100. If it is assumed that h= 0.1, 
then performance degradation is 2 to 6 percent. For h= 0.1, 
0.22 mega-byte/second per subsystem is needed. Thus, if a 
1 mega-byte/sec bus is provided, then a system, made up of 


fU)= 


H>(i-H>y 1 
1 (1 w ) 5 ’ 


0<w^l. 


(5) 


(2) Each subsystem is identified with the number k, 
l^k^n. The A:th subsystem is denoted by S k . The 
correspondence of to S k is assumed to be random. 
Namely, S k corresponds to S tj with equal probability 
1 /n. 

(3) The request Ri s processing time by subsystem is 
denoted by 

P 

L w,3 • 

Since the system is symmetric and, if a page transfer 
time between subsystems is assumed independent of 


TABLE I.—Performance Degradation 


h 

c 

(mega- 

byte/sec) 

P = 

1/3 

P= 

1/10 

P= 

1/100 

P= 

1/1000 

E[S]/E[T ] E[S]/E[T] 

= 1/20 =1/200 

1/20 

1/200 

1/20 

1/200 

1/20 

1/200 

0.002 

11 

0.998 

0.998 

0.999 

0.998 

1.000 

0.999 

1.000 

1.000 

0.02 

1.1 

0.982 

0.981 

0.987 

0.981 

1.000 

0.987 

1.000 

0.997 

0.04 

0.55 

0.966 

0.962 

0.974 

0.963 

0.993 

0.974 

0.999 

0.993 

0.06 

0.37 

0.950 

0.944 

0.962 

0.946 

0.990 

0.962 

0.999 

0.990 

0.08 

0.28 

0.935 

0.927 

0.949 

0.929 

0.987 

0.949 

0.998 

0.987 

0.1 

0.22 

0.920 

0.910 

0.938 

0.913 

0.984 

0.038 

0.998 

0.984 

0.15 

0.15 

0.885 

0.871 

0.909 

0.875 

0.976 

0.909 

0.997 

0.976 

0.2 

0.11 

0.852 

0.835 

0.882 

0.840 

0.968 

0.882 

0.996 

0.968 

0.3 

0.073 

0.793 

0.772 

0.833 

0.778 

0.952 

0.833 

0.994 

0.952 

0.5 

0.044 

0.697 

0.670 

0.750 

0.677 

0.923 

0.750 

0.990 

0.923 

0.7 

0.031 

0.622 

0.592 

0.682 

0.600 

0.900 

0.682 

0.986 

0.8% 

1.0 

0.022 

0.535 

0.504 

0.600 

0.512 

0.857 

0.600 

0.981 

0.857 

1.5 

0.015 

0.434 

0.404 

0.500 

0.412 

0.800 

0.500 

0.971 

0.800 

2.0 

0.011 

0.365 

0.337 

0.429 

0.344 

0.750 

0.429 

0.962 

0.750 

3.0 

0.0073 

0.277 

0.253 

0.333 

0.259 

0.667 

0.333 

0.944 

0.667 

5.0 

0.0044 

0.187 

0.169 

0.231 

0.174 

0.545 

0.231 

0.911 

0.545 
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TABLE II.—Overhead 


w 

7=1 

7=2 

''t 

II 

'“-I 

7=6 

0.9 

0.1 

0.505 

0.750 

0.833 

0.7 

0.3 

0.545 

0.752 

0.833 


the subsystems chosen, Eq. (3) can be used as PwJ 
where r is to be replaced with /( j) of Eq. (3). 

P w j={pE[T]+p(l-f( j))(E[U]+E[V])+E[S]}/q+E[A] 

The following observations are made. 

(1) Under the all selection policy, each request is equally 
selected by a subsystem. Thus, the average service 
time would be 


n 

For a system made up of a single subsystem, service time 
is P t j, which is described by Eq. (2). The system load would 
be very well balanced, because each request is assigned to 
the first subsystem that becomes empty. The assignment is 
a reversed FIFO. 

(2) Under the sufficient selection policy, only k subsys¬ 
tems become candidates for selections, where 1^ j=k 
and k is determined by the policy. Variable k itself is 
a random variable. The average service time would be 

y k p 

r »j 

k 

Under this policy, the load would also be well bal¬ 
anced but there may be some processing time loss, 
because there are some chances that empty subsys¬ 
tems cannot take unprocessed requests because they 
are assigned to some other busy subsystems. However 


this loss is ignored for simplicity and the necessary 
intersubsystem bus capacity is calculated for k=l, 2, 
4 and 6, whose results are shown in Table III. It is 
assumed that the system consists of six subsystems. 
In Table II, the ratio 

s;„ a 


is shown. The above ratio e is related to variable h 
defined by Eq. (4) as 

h=e{E[U]+E[V])/E[T ]. 

From Table III it can be observed that, for p=0.01 and 
E[S]/E[T\= 1/200, the system can work with 10 percent 
degradation with /i=0.15 or c =0.247 for j =2 and w=0.9. 
Thus with an intersubsystem bus whose capacity is 1 mega¬ 
byte/sec, the system of four can be constructed. With a bus 
of 4 mega-byte/sec, a system of 16 can be built. If E[S]/ 
E[T]= 1/20 can be assumed, h= 0.900 or C=0.0529 is suffi¬ 
cient. This allows 19 subsystems for a 2 mega-byte/sec bus. 

It should be noted that the above analysis is very simple. 
A more elaborate and precise analysis is clearly necessary 
and is now under way. Its results will be reported elsewhere. 
Relating to this, there are better subsystem selection poli¬ 
cies. One of them is, for instance, to use the following 
criterion: 

(a) Compute vv, using Eq. (1) 

(b) If w is fairly high for a subsystem, say w>0.7, then 
any subsystem whose value of w is not greater than 
0.7 drops out. 

(c) The smaller the value of w is, the more subsystems 
can be assigned. 

(d) When w=l/n, assign all the subsystems. Analysis of 
this policy would require a more elaborate analysis 
and is not included in this paper. 


TABLE III.—Intersubsystem Bus Capacity 


h 



3 

II 

o 

V© 




II 

© 

u> 


7=1 

7=2 

7=4 

7=6 

7=1 

7=2 

7=4 

7=6 

0.002 

3.665 

18.5 

27.5 

30.6 

11.0 

20.0 

27.6 

30.6 

0.02 

0.367 

1.85 

2.75 

3.06 

1.10 

2.00 

2.76 

3.06 

0.04 

0.183 

0.925 

1.38 

1.53 

0.55 

1.00 

1.38 

1.53 

0.06 

0.122 

0.617 

0.917 

1.02 

0.367 

0.667 

0.918 

1.01 

0.08 

0.0916 

0.463 

0.688 

0.764 

0.275 

0.500 

0.689 

0.764 

0.1 

0.0733 

0.370 

0.550 

0.611 

0.220 

0.400 

0.551 

0.611 

0.15 

0.0489 

0.247 

0.367 

0.407 

0.147 

0.267 

0.367 

0.407 

0.2 

0.0367 

0.185 

0.275 

0.306 

0.110 

0.200 

0.276 

0.306 

0.3 

0.0244 

0.123 

0.183 

0.204 

0.0733 

0.133 

0.184 

0.204 

0.5 

0.0147 

0.0740 

0.110 

0.122 

0.0440 

0.0800 

0.110 

0.122 

0.7 

0.0105 

0.0529 

0.0786 

0.0873 

0.0314 

0.0571 

0.0787 

0.0873 

1.0 

0,00733 

0.0370 

0.0550 

0.0611 

0.0220 

0.0400 

0.0551 

0.0611 

1.5 

0.00489 

0.0427 

0.0367 

0.0407 

0.0147 

0.0267 

0.0367 

0.0407 

2.0 

0.00367 

0.0185 

0.0275 

0.0306 

0.0110 

0.0200 

0.0276 

0.0306 

3.0 

0.00244 

0.0123 

0.0183 

0.0204 

0.00733 

0.0133 

0.0184 

0.0204 

5.0 

0.00147 

0.00740 

0.0110 

0.0122 

0.00440 

0.00800 

0.0110 

0.0122 
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CONCLUDING REMARKS 

Extensibility is one of the important system properties of 
future systems. Among many extensibility aspects, the pres¬ 
ent work is concerned with extensibility in capacity. Exten¬ 
sibility, in this sense, is first defined and then a scheme to 
obtain it is proposed and described. The scheme is shown 
as an algorithm and then its correctness is verified. Exten¬ 
sibility contains automatic load sharing. Scheme optimality 
is also shown. Performance analysis shows that this scheme 
can be used to build a fairly large size extensible data base 
system without much performance degradation. 

Extensibility in capacity means three things; physical ex¬ 
tensibility, logical extensibility and minimal performance 
degradation. The key property of the scheme for logical 
extensibility is that each request is broadcast to each sub¬ 
system and is then tried by every subsystem. Eventually, 
however, one and only one subsystem completes the re¬ 
quest. Each subsystem is autonomous and it itself deter¬ 
mines when to drop out. In doing this, no exchanges of 
information about the load of each system are required. For 
this reason, no consistency problems occur. No system wide 
mutual exclusions are necessary in this respect, which 
makes the logic considerably simpler. 

The system performance is analyzed using a simple mean- 
value analysis. It is observed, from this analysis, that a fairly 
large size data base system can be built, based on the pro¬ 
posed scheme. Necessary capacity of the intersubsystem 
bus can remain realistic. Any means of reducing missing 
page ratio would greatly help improve the performance of 
the system. Here, magnetic bubble or CCD memory can be 
a great help. 
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Practical problems in a distributed application 
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INTRODUCTION 

In this report the term “distributed application” is used to 
refer to application programs with modules which operate 
on two or more computers, and which communicate with 
each other in real time. In particular, this report describes 
experience with an IBM Program Product, Trend Analysis/ 
370 (TA/370). TA/370 is a distributed application employing 
two computers, a host and a satellite. The satellite computer 
is used for a color graphics feature, which can produce line 
graphs or bar charts in multiple colors. The host computer 
is used for data base management and non-graphical data 
transformations (such as computing ratios). Figure 1 shows 
the TA/370 system configuration, and details on the system 
can be found in the reference manuals. 5,6 In TA/370, user 
queries are processed in the host computer. If the query 
contains a request for a graph or bar chart, the appropriate 
parameters (data and formatting) are transmitted from the 
host to the satellite. The satellite acts as a formatter and 
controller for the graphics terminal on which the graphs and 
bar charts are displayed. Thus, only the graphics formatting 
and control functions in TA/370 are distributed, and com¬ 
munication is from the host to the satellite. 

Of the many possible reasons for a distributed application, 
the reason for distributing the graphics functions in TA/370 
appears to be performance. Specialized graphics formatting 
functions, such as labelling and scaling, can be performed 
in the satellite based on encoded commands from the host. 
Thus, not all the formatting data for a graph need be trans¬ 
mitted from the host to the satellite (reducing transmission 
time) and the formatting commands can be executed in the 
satellite. Reduced transmission times and a special-purpose 
graphics formatter are intended to improve response time 
and reduce costs for creating graphical outputs in TA/370. 
To achieve these benefits, a special-purpose protocol for 
data communications was developed, and the satellite pro¬ 
grams for graphics formatting were application-specific and 
coded in assembler language. 

Our experience with TA/370 began with an attempt to 
modify the code for the satellite in order to be able to use 
different i/o hardware than supported by TA/370. These 
were differences in implementation and not in functional 
capability. Because of the special-purpose nature of the sat¬ 
ellite code and data communications protocol, we found 


modification difficult. Attempting to add new graphical func¬ 
tions to TA/370 appeared even more complicated. The dif¬ 
ficulty in modifying and extending the graphical capabilities 
of TA/370 led us tb investigate the possibility of using a 
general-purpose graphical subroutine package, developed by 
IBM Research for other applications, 2 in TA/370. Because 
this subroutine package was developed to operate on the 
host computer, we expected that we would lose the per¬ 
formance advantages of the distributed processing for graph¬ 
ics in TA/370. 

We implemented a prototype revision of TA/370 using the 
host-based graphics subroutine package. In comparing the 
prototype with the original TA/370, there was no noticeable 
change in response times. Further investigation of the two 
versions was made to find out why the theoretical perform¬ 
ance advantage for distributed processing was not realized 
in practice. The results of this investigation indicated that 
the overheacf in the host-satellite protocol was so large that 
advantages from reduced data transmission or parallel proc¬ 
essing did not affect response time significantly. In addition, 
advantages from distributed processing appeared to be offset 
by need for data encoding and decoding to communicate 
between the host and the satellite. Use of a general-purpose 
software package operating on the host computer seemed to 
offer greater flexibility, easier maintenance and extensions, 
and reduced hardware and programming costs without any 
performance penalty compared to the original version of 
TA/370. 

In the next section of the report we describe in more detail 
why we wanted to modify TA/370 and the problems we 
encountered in doing so. The third section describes the 
implementation of the “non-distributed” prototype and the 
benchmark comparison with the distributed TA/370. In the 
fourth section we analyze the results of the comparison, and 
in the fifth section we discuss some alternatives for distrib¬ 
uted processing in applications such as TA/370. 

THE PROBLEMS WITH THE DISTRIBUTED 

VERSION 

We were interested in modifying TA/370 for two reasons: 
(1) to make it operational on a satellite computer with a 
different graphics terminal and host-satellite communica¬ 
tions interface, and (2) to add new graphical functions (such 
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• Menus • Report Labels • Time Series 

• Menu Processing and Titles Data Values 

Instructions 

Figure 1—Trend analysis/370 configuration 

as drawing maps). The modification to support different I/O 
hardware involved adding a few hundred lines of code to the 
TA/370 modules in the satellite. The addition of new function 
would require adding modules in both the satellite and the 
host. Because of time constraints and the difficulty of mod¬ 
ifications, only the modifications needed to support the I/O 
hardware were completed. 

In making the modifications we encountered several prob¬ 
lems: 

1. The satellite code was special-purpose, designed for 
this application and satellite computer, and was written 
in assembler language. Even after six months of work, 
we never totally understood the satellite code, and the 
programmer who wrote the original code had to make 
the modifications. Even if we had been experts in the 
assembler language for the satellite, modifying the code 
would have been difficult because it required detailed 
knowledge of the hardware and of the data communi¬ 
cations protocols. 

2. Communication between the host and the satellite used 
a special-purpose set of records, with five basic record 
types, each of which was encoded at the bit level. That 
is, instead of using an extension of a standard com¬ 
munications format, such as the ASCII character set, 
TA/370 essentially used five special-purpose bit strings 
for which we had to learn the encoding and decoding. 

3. The distribution of graphics function had resulted in 
duplication of functions in the host and the satellite 
computers. Specifically, the host modules scaled the 


data to be displayed into value ranges compatible with 
the range of values which could be represented in one 
word on the satellite, and the satellite rescaled the data 
into ranges compatible with the address space of the 
graphics display. In addition, the host modules en¬ 
coded the graphics commands and data into the five 
special purpose records for transmission to the satel¬ 
lite, and the satellite modules decoded these records 
and then recoded the data for transmission to the 
graphics terminal. Thus to make any modifications to 
the graphics modules, one had to understand the details 
of both the host and the satellite coding schemes. 

In looking at more complicated modifications of TA/370, 
such as adding new graphics functions or supporting a dif¬ 
ferent type of graphics terminal, it was apparent to us that 
problems, such as those listed above, would become more 
severe. We would have to add and modify code in both the 
host and the satellite and we would have to modify or extend 
the data communications records. 

EXPERIENCE WITH A NONDISTRIBUTED VERSION 

Graphics functions in the nondistributed version 

We had several years experience with an application sim¬ 
ilar to TA/370 in which the graphics functions were not 
distributed. 2 This system used a general-purpose graphics 
subroutine package, called DISPLIB, which was developed 
to support a variety of graphics functions and graphics ter¬ 
minals. This package, written in FORTRAN, provides basic 
commands for displaying text, lines, and points, as well as 
for display transformation (e.g., scaling) and user input. The 
package is similar in style to the standard subroutine pack¬ 
age currently being proposed by ACM SIGGRAPH. 1 The 
package also uses extensions of the ASCII or Correspond¬ 
ence character sets for graphics communication to the ter¬ 
minal; this convention is used by several terminal manufac¬ 
turers. Because the package operated on the host computer, 
its use in TA/370 required that we not distribute the graphics 
functions. We expected that this change to a nondistributed 
application might degrade performance. 

A prototype version of TA/370 using DISPLIB was im¬ 
plemented. The prototype supported only simple bar charts 
and therefore was not functionally equivalent to the original 
TA/370. In addition, the prototype simply translated the five 
records (that TA/370 created to send to the satellite) into 
DISPLIB calls. It would have been more efficient to replace 
the TA/370 code that creates the five records with code 
calling DISPLIB, but simply translating the records required 
fewer changes to TA/370 and was easier to program. 

Comparison: distributed and nondistributed 

Table I gives a comparison of the graphics functions in 
TA/370 and those in our prototype revision of TA/370. This 
comparison indicates the basic differences between the two 
versions. 

The prototype was implemented and a benchmark bar 
chart containing data on six organizations for five years 
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TABLE I.—Comparison of Graphics in TA/370 and TA/370 Prototype 


Feature 

TA/370 

TA/370 Prototype 

graphics package 



language 

assembler 

FORTRAN 

style 

special-purpose 

general-purpose 

graphics function 



implemented on 

satellite 

host 

graphics functions 

building 


invoked by 

graphics records 

subroutine calls 

host code 



language 

PL/1 

PL/1 

type of graphics 

line graphs and bar charts 

simple bar charts 

communications 

5 special record types 

ASCII extension 

what communicated 

data + graphics 

data + graphics 


format codes 

formats 

terminals per satellite 

4 

1 

terminal types supported 

1 (raster) 

3 (raster, vector, storage) 

size of host 

29K bytes 

113K bytes 

code for graphics 



(approximate) 



lines of host code written 

500 

170 

by applications programmer (approx.) 



satellite code size 

47K bytes 

24K bytes 


(approximate) 


(=30 bars) was used to compare performance. Amount of 
code and communication load also were compared, and flex¬ 
ibility and programming cost were compared subjectively. 

1. Response time: For each version the time to draw the 
bar chart was about 15 seconds. (Note: in other in¬ 
stallations of TA/370 which use a S/7 with hardware 
features to assist in host-satellite communications, re¬ 
sponse times for bar charts similar to the one used in 
our benchmark are about 10 seconds.) 

2. Communication load: In the original TA/370 imple¬ 
mentation about 800 bytes of bar chart values and 
formatting keys were transmitted from the host to the 
satellite. In the prototype version of TA/370 about 2500 
bytes of bar chart values and formatting data had to be 
transmitted. Transmission was at 1200 Baud. TA/370 
sends 268 byte blocks and the prototype used 256 byte 
blocks. 

3. Communications protocol: Both versions use 1200 
Baud asynchronous communication over voice grade 
lines. The original version uses a transmission code 
(Paper Tape Transmission Code) developed for IBM 
2740-1 terminals. The prototype based on DISPLIB 
uses ASCII or Correspondence codes. The protocol in 
the original version uses five special-purpose data com¬ 
munications records with host-satellite “handshaking” 
after each block is transmitted. The prototype uses 
“standard” ASCII codes with “handshaking” only at 
the start of host-satellite communications. 

4. Amount of code: Even though we implemented only a 
subset of the TA/370 graphics functions, the prototype 
version of TA/370 required much more code in the host 
because a general-purpose package was used. How¬ 


ever, the applications programmer wrote less code in 
the prototype because this package was available. The 
code written by the applications programmer in the 
prototype is about 10 percent of the total amount of 
code, whereas in the original TA/370 it is 100 percent. 
We estimate that if we had chosen to implement only 
the TA/370 graphics functions (eliminating any DIS¬ 
PLIB code not required for these functions) and to 
rewrite the TA/370 host code for (creating graphics 
records rather than adding code to translate those rec¬ 
ords into DISPLIB calls), the amount of host code in 
both versions would have been approximately equal. 
Any implementation of the prototype would require 
less code in the satellite. Moreover, the satellite code 
used in the prototype would support all the TA/370 
graphics functions and would not require modifications 
for most additional graphics functions. 

5. Size of satellite computer: The original TA/370 required 
an IBM System/7 with disk storage, while the proto¬ 
type could operate using an IBM 5100. The System/7 
is about five times more expensive that the 5100 (about 
$50,000 more). 

6. Programming cost and skill level: Because the proto¬ 
type used a high-level language of graphics calls rather 
than having to create records at the bit level, we believe 
that the cost to program the prototype was less than 
the cost to program the equivalent function in the orig¬ 
inal TA/370. The prototype was developed in less than 
two person months for a programmer who was not 
familiar with the DISPLIB package. In addition, both 
the host and satellite code for graphics in the prototype 
would not need to be changed if new graphics functions 
were required. This type of general-purpose graphics 
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would greatly reduce the programming cost for modi¬ 
fications of TA/370. 

7. Flexibility: Because the prototype used a high-level 
language for graphics functions, and a general-purpose 
graphics subroutine package which supported several 
types of terminals, we believe that the prototype ver¬ 
sion was more flexible and would be easier to modify 
or extend than the original version of TA/370. 

In summary, the experience with the prototype indicated 
that a nondistributed version of TA/370, using a general- 
purpose graphics subroutine package would not degrade per¬ 
formance, even though the communication load was in¬ 
creased, and would offer significant advantages in terms of 
hardware costs, programming costs, and flexibility. The per¬ 
formance results of the benchmark were not as expected, 
so the prototype and original versions were further analyzed 
to try to better understand the performance implications of 
the distributed versus the nondistributed implementations. 


ANALYSIS: DISTRIBUTED VERSUS 

NONDISTRIBUTED VERSIONS 

One could argue that the prototype performed as well as 
the original version of TA/370 because we implemented only 
a subset of the graphics functions of TA/370 and because we 
had the advantage having the original version as a model for 
implementing the prototype. Both these arguments are par¬ 
tially true, but further analysis indicated that even with equal 
functions and equal skill in implementation one should not 
expect much difference in performance between the distrib¬ 
uted and nondistributed versions. The analysis indicated 
that the predominate factor in performance was the time to 
transmit data from the host to the satellite over the 1200 
Baud line. For the bar chart used in the benchmark com¬ 
parison the original version of TA/370 transmitted less 
(about one-third as much) data than the prototype version. 
However, in the original version of TA/370 there is “hand¬ 
shaking” between the host and the satellite once for every 
transmitted block. This “handshaking” is implemented in 
software. The “handshaking” consumes enough time to 
eliminate any performance advantage for the original TAJ 
370 due to reduced data transmission. 

This informal analysis of the performance of each version 
is substantiated by an analysis using a simple model. This 
model is a simplification of a model of distributed (graphics) 
systems proposed by Foley. 3 The model consists of four 
factors which affect performance: data base access time in 
the host, execution time in the host, execution time in the 
satellite, and data transmission time. Thus for any graphics 
function in either version, a model of performance is: 

T=TD*ND+TH*IH+TS*IS+TT*TL 

where: 

T=total time (e.g., response time) 

TD=average time for data access in host, ND=number of 

data base accesses 


TH=execution speed of host, TS=execution speed of sat¬ 
ellite 

IH=instructions executed in host, IS=instructions executed 
in satellite 

TTtransmission speed, TL=transmission load 

For either the distributed or nondistributed versions of TA/ 
370, the TT*TL term in the model is the predominate factor. 
Transmission speed is 1200 Baud and the transmission load 
is in thousands of bits, so that TT*TL results in times meas¬ 
ured in seconds. The data base access times (TD*ND) in 
both versions are equal because the same data must be 
retrieved. These times are usually in hundredths of a second. 
The host and satellite execution speeds are approximately 
equal because the satellite is about one-third as fast as the 
host, but the host is time-shared and time-sharing, on the 
average, reduces the effective execution speed by about 
one-third. The execution plus data base access times 
(TD*ND+TH*IH+TS*IS) in both versions of the system 
are in tenths of seconds. Thus, the model indicates that any 
performance difference between the two versions is going 
to be due to differences in transmission times. Detailed 
expressions for computing the transmission loads in each 
version were worked out, but are omitted here. In general, 
for a simple graph or bar chart the transmission load in the 
original TA/370 is about half that in the prototype (approx¬ 
imately 600 bytes compared to 1200 bytes). For a compli¬ 
cated graph it may be only one-sixth as much (about 1000 
bytes compared to 6000 bytes). Note that at 1200 Baud the 
transmission load in the prototype version could take up to 
40 seconds to transmit. In practice, however, the perform¬ 
ance advantage that this simple model predicts for the orig¬ 
inal version is not realized. As previously mentioned, the 
practical failure to realize this advantage seems to be due to 
the overhead of the transmission protocol used in the orig¬ 
inal version. 

ALTERNATIVES FOR DISTRIBUTED FUNCTION 


Models of distributed applications 

To choose among alternative configurations for a distrib¬ 
uted application such as TA/370, one could use the model 
proposed by Foley. 3 Given a set of alternatives for host, 
satellite, and data link hardware this model can be used to 
select the configuration which will produce the best response 
time for a given cost constraint. However, one needs to 
have predetermined the distribution of function and to have 
estimated the frequencies of graphic macro instructions, the 
probabilities of various user interactions, and the data base 
accesses required per interaction. 

It would be very difficult to estimate the parameters of 
TA/370 and the parameters of hardware alternatives needed 
for the Foley model. More significantly, the hardware alter¬ 
natives for TA/370 are restricted by the application’s re¬ 
quirement for a color graphics terminal which could be at¬ 
tached to a host over voice grade lines. Therefore, the Foley 
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model does not provide much help in selecting among alter¬ 
native distributions of function for TA/370. However, Foley 
used his model to analyze four “typical” graphics applica¬ 
tions, one of which, two dimensional drawing, is similar to 
TA/370. 3 Foley’s analysis considered over 12,000 possible 
hardware configurations. Based on this analysis, Foley rec¬ 
ommended that, in general, the best system configuration 
for graphics applications consisted of the simplest possible 
satellite (which provided enough function) using a voice- 
grade (2400 Baud) data link. Foley’s analysis indicated that 
the data link and bulk storage on the satellite are the two 
components which most influence response time. Foley’s 
analysis of distributed applications thus is very similar to 
our study of TA/370: data transmission load is the key var¬ 
iable affecting cost/performance in a distributed application, 
and distribution of function (or data) is useful only to the 
extent that it helps reduce this load. 

Foley’s model gives us insight in selecting a hardware 
configuration, given a set of functional modules for a dis¬ 
tributed application. For a given hardware configuration and 
function modularization, a graph partitioning model, such as 
that suggested by Hamlin, 4 can be used to decide which 
modules should be distributed. In a graph model of a pro¬ 
gram, the modules are nodes and among the nodes there are 
directed arcs which represent the calling and returning struc¬ 
ture of the program. Associated with each arc is a cost, such 
as the amount of data passed or the overhead time for 
calling. Other costs, such as the amount of global data ac¬ 
cessed from each module or the execution time for each 
module on the host and on the satellite, may be associated 
with each node. The graph partitioning algorithm attempts 
to minimize the cost associated with assigning the modules 
(nodes) to the host or the satellite. To do this, the model 
requires an estimate of the frequencies of data accesses and 
inter-module calls. For an example of such a model see 
Hamlin. 4 As indicated by Hamlin’s example, unless there 
are constraints (such as limited memory space) which pre¬ 
vent all modules from being assigned to one computer, the 
cost function usually is minimized if there is no distribution 
of function. 

Note that both the Foley and Hamlin models assume that 
the program modules are defined. Yet the definition of the 
modules is a primary factor affecting the cost functions in 
both models. This suggests the need for a model based on 
the costs of performing various graphics functions, inde¬ 
pendent of program modularization, which could be used to 
analyze alternative modularizations. In addition, both 
models require many parameters of the application, pro¬ 
gram, and hardware. Estimating these parameters is diffi¬ 
cult, and likely to be error prone. To the extent that the 
models are sensitive to errors in the parameters one would 
expect the models to suggest suboptimal alternatives which, 
if used, would need performance tuning. For performance 
tuning, it would be useful to be able to move program mod¬ 
ules between the host and satellite computers. To make this 
easier, one needs to be able to compile and execute the 
same language on both machines, 4 and to have a mechanism 
which supports calls and parameter passing among modules 
on different machines. 


Possible alternatives for TA/370 

The general guidelines of the Foley and Hamlin models 
for distributed applications suggest two alternatives for TA/ 
370. 

1. Reduce the size of the satellite computer and the mod¬ 
ules assigned to it to as small as possible to contain the 
bar chart and graph formatting functions. Place all re¬ 
maining modules in the host. This distribution will limit 
data transmission to data values only. The double en¬ 
coding/decoding and data scaling currently in TA/370 
will be eliminated. Our prototype is representative of 
this alternative. 

2. Distribute the entire application program to the satel¬ 
lite, with batch transmission of a subset of the data 
base to each satellite on an as-needed basis. However, 
both the data base and the application program are too 
large for most satellites. A variation of this alternative 
would be to keep a working-set of the data base in the 
satellite (e.g., the data used in the previous “n” dis¬ 
plays), and distribute all the graphics functions to the 
satellite, as done in the original TA/370. This alterna¬ 
tive will reduce the data transmission because all the 
data for a graph or chart will not need to be transmitted 
if it is a variation on one of the charts stored in the 
working-set. 

For either alternative it would be beneficial to use the 
highest speed communication link compatible with the ap¬ 
plications’ requirement for a voice-grade line. Currently 
4800 Baud is possible, although to use 4800 Baud would 
require changes to the satellite hardware (System 7 or 5100) 
and to the communications protocol (bisynchronous instead 
of asynchronous). 

CONCLUSIONS 

The experience with TA/370 identified several practical 
problems in distributed applications: 

1. The potential performance advantage of distributed 
function was reduced because of the overhead of host- 
satellite communication protocol. 

2. Distributed function resulted in special-purpose pro¬ 
gramming and data encodings which made modifica¬ 
tions and extensions difficult. 

3. Distributed function greatly increased the cost of the 
satellite computer, without compensating savings in 
the cost of the host computer. 

4. Distributed function probably increased the amount of 
code and the programming cost for this application. 

5. Transmission time between the host and the satellite 
was the primary performance factor. The application’s 
need for a 1200 Baud line limited the performance 
improvement that could be achieved by distributed 
function as long as the data base was stored on the 
host. 
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6. Deciding how to distribute function involves consid¬ 
ering hardware configuration, alternative distribution 
of program modules and data, and alternative modu¬ 
larizations. Existing analytic models of distributed ap¬ 
plications do not encompass all of these considerations. 

7. In designing distributed applications such as TA/370 
the constraints are: the available hardware, the hard¬ 
ware costs, and the software development and main¬ 
tenance costs. As with most programs, the software 
development and maintenance costs are likely to be 
the most important constraint. 

This study does not indicate that distributed processing is 
ineffective for applications such as TA/370. It does indicate 
that distributed processing is not simply a separation of 
function among computers, because the separation requires 
communication. Because communication is involved, design 
decisions can become more complicated (as indicated by 
Foley’s model), and unexpected inefficiencies can result 
(such as the “handshaking” problem). Most importantly, 
when the design objectives change (e.g., added function or 
different hardware), a distributed implementation may be 


more difficult to modify than a non-distributed version. Our 
experience with TA/370 indicates that these “practical” 
problems can eliminate a theoretical advantage for a distrib¬ 
uted application. 
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A panel session—Distributed systems—A challenge to 
management 


SESSION CHAIRMAN—GEORGE M. CRANDELL, JR. 

McKinsey & Company, Inc. 

Panel Members 

Lester Stubbs—Mattel Toys 

Michael C. Dowling—Fireman’s Fund Insurance Company 
Mario Calderin—Aratex, Inc. 

PANEL OVERVIEW—George M. Crandell, Jr. 

This session addresses the management issues and deci¬ 
sions that concern MIS directors, project leaders, and users 
in the implementation of distributed systems. The panelists, 
who have been selected to provide a cross section of indus¬ 
tries, will briefly describe their distributed systems and then 
focus on the most significant managerial/organizational chal¬ 
lenges encountered during implementation. The pros and 
cons of alternative approaches will be evaluated, and mis¬ 
takes as well as successes will be discussed. 

Areas to be explored include: 

1. Considerations and caveats in allocating responsibili¬ 
ties between headquarters and local MIS personnel, 
users, hardware vendors, and software houses/con¬ 
sultants 

2. Approaches for carrying out system design, develop¬ 
ment, implementation, training, maintenance and trou¬ 
ble handling 

3. Lessons learned by the panelists from their experi¬ 
ences, and their suggestions for avoiding potential pit- 
falls. 

There will be considerable time for questions and general 
discussion following the presentation. 


INSTALLING IBM 3790’s IN A MANUFACTURING 
ENVIRONMENT—Lester Stubbs 

During the last two years Mattel Toys has installed various 
phases of its new manufacturing system utilizing the IBM 
3790 in a distributed processing environment. Each instal¬ 
lation has been extremely smooth and relatively error-free. 
Besides having luck on its side, Mattel used a project man¬ 
agement approach throughout the development and imple¬ 


mentation process that contributed greatly to this success. 
This presentation is centered around the actual implemen¬ 
tations and the planning that went into them. 

There are two extremely important factors that must be 
continually evaluated for any new system being developed. 
The first is constant review to make sure that the system 
will meet the end-users’ needs and that the end-users can 
effectively work with the system in a production environ¬ 
ment. The second factor is selecting and tailoring the equip¬ 
ment so that it will perform properly when in production. A 
closer look at how Mattel approached these two factors will 
be presented. 

As installation nears, there is system testing, user testing, 
acceptance testing, system documentation, equipment in¬ 
stallation and conversion. How Mattel approached each of 
these tasks will be reviewed. The project management ap¬ 
proach used will be emphasized and the critical factors will 
be evaluated. 

Finally, there is the day of installation. If properly 
planned, this event can be very anti-climactic. The first few 
weeks of production will be reviewed with emphasis on 
Mattel’s warranty period concept. 

Everyone hopes to install a system that is a glowing suc¬ 
cess, and, through personal experiences and the experiences 
of others, system installations can result in rewarding ex¬ 
periences for all. Hopefully, this presentation, and others 
like it, will help. 


DISTRIBUTED DATA PROCESSING IN A CLERICAL 

ENVIRONMENT—Michael C. Dowling 

Fireman’s Fund direction and plans in distributed data 
processing must be taken in the context of the nature of its 
business, its field organization, its current data network, its 
relationship to American Express Data Processing, and its 
view of the business process and its role in the marketplace 
over the next 10 years. Given our structure and business 
needs we sought a means of reducing our operating expenses 
while improving the quality and speed of services we pro¬ 
vide, i.e., insurance policies, quotes, and loss payments. 
Data processing and improved manual procedures provided 
a basis for achieving this business objective. 

In the data processing area, we looked at the technology 
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available and found that three fundamentally different ap¬ 
proaches were available: 

1. Batch store and forward 

2. On-line processing 

3. Local processing. 

Our research into the business needs indicated that to 
reduce cost and improve service we should use all three 
approaches in an optimum mix. Thus we are aggressively 
pursuing the development of systems utilizing all three fun¬ 
damental approaches. From a strategic point of view, we 
intend to place computing power in the hands of our local 
offices while retaining full control of programming and pro¬ 
gram version release at the central site. 

In order to adequately address the new business and sys¬ 
tems environment of remote computers and computer proc¬ 
essing in essentially clerical offices, we established a Dis¬ 
tributed Systems Department. This department is responsible 
for all local applications and the local portion of batch store 
and forward applications. The department reports to the 
Senior Systems Executive and is essentially a microcosm of 
the total systems organization. 

Not only did the systems organization undergo substantial 
revision, but new user departments were created and an 
extremely active and important executive steering commit¬ 
tee was established. The business executive was made re¬ 
sponsible for priority setting, business definition, systems 
justification and establishment of a responsive Business Sys¬ 
tems Plan. 

As our systems become an integral part of our daily field 
operations, we are learning what manufacturing companies 
with computer assisted production have known for some 
time: that a new level of responsiveness to system problems 
is required. We have been deeply committed to this task for 
the last two years and we have learned lessons in every area 
from equipment selection to field implementation. Most im¬ 
portantly, however, we have developed from our experience 
seven postulates for effective distributed system develop¬ 
ment: 

1. Demand user participation 

2. Insure management commitment 

3. Maintain high visibility 

4. Fund development on a step by step basis 

5. Define accountability 

6. Balance technology with business needs 

7. Integrate manual systems planning with EDP systems 
planning. 


DISTRIBUTED SYSTEMS IN RETROSPECT: 

LESSONS WE LEARNED THE HARD WAY—Mario 

Calderin 

Aratex, Inc., an Encino, California based textile rental 
company, installed one of the first 3790 distributed process¬ 
ing networks in the country. Running under the Systems 
Network Architecture (SNA), the 3790 has brought new 
data processing capabilities to the company’s remote loca¬ 
tions. 

At each of Aratex’s 35 locations across the country, plant 
personnel using 3271 display terminals linked to a 3791 con¬ 
troller, which acts like a small computer with its own locally 
stored data, constantly update and interrogate data regarding 
Customer Billing, Accounts Receivable, Payroll, Sales Anal¬ 
ysis, and Inventory Control. All data is checked for validity 
using extensive edit programs. Since the system is installed 
at locations where there is no EDP expertise, all transactions 
are checkpointed to provide automatic restart capability. 

At the end of the day, all pertinent data is transmitted 
unattended to a 370/148 at the Encino National Data Center. 
After the necessary processing is performed, approximately 
500,000 lines of output is transmitted back to the remote 
locations daily, where it is printed on the 3790 printer. A 
3790 system is also installed as part of the network at the 
company’s Central Warehouse facility in Fresno, California. 
The system accepts on-line order entry transactions from 
3790’s at any plant in the network and produces order status, 
packing slips, shipping confirmation, and invoices at the 
warehouse, as well as assigning goods for replenishing ran¬ 
dom locations, and regulating the warehouse work schedule 
based on order arrival frequency. 

The entire system is comprised of over 400 programs 
developed over the last two years. As one of the first com¬ 
panies in the United States to deal extensively with distrib¬ 
uted processing, we have developed expertise in the control 
aspects. Our experience indicates that the most important 
areas to consider are: 

1. Control of programs at remote sites 

2. Data synchronization between remote sites and central 
data bases 

3. Communication programs to allow simultaneous trans¬ 
missions to and from the host system by multiple users 

4. User backup and recovery in case of down-time 

5. Response to user questions while operating the system 

6. Centralized problem determination for both hardware 
and software 

7. User training 

8. System installation guidelines. 




Area Director 
Charles W. Bachman 
Honeywell Information Systems 
Billerica, Massachusetts 


Database management systems 


Database management as a technology has made tremendous strides in recent 
years. Database management systems are offered by every major hardware ven¬ 
dor and many software houses. Installations which operate database management 
systems in support of their management’s objects now number in the thousands. 
Two database management specialized conferences are held each year with 
hundreds of attendees present. The first of these is held by the ACM Special 
Interest Group on the Management of Data (SIGMOD). The second is the Very 
Large Database Conference (VLDB) which has moved around the world with 
Germany as the host for 1978. These newer conferences have collected most of 
the leading edge database technology papers because it is possible to concentrate 
more of the research-oriented people, their papers and discussions into smaller 
towns and smaller hotels. 

Therefore, the Database Management sessions for NCC ‘78 have focussed 
upon the problems and solutions associated with use of the currently available 
database management systems. Four panel discussions and four technical ses¬ 
sions have been planned for your attendance at NCC ‘78. Three of these panel 
sessions are focussed upon users’ experiences relating to their database systems. 
One panel session focusses on “end user facilities,” one of the leading edge 
areas of database management. The four technical sessions bring papers which 
are oriented toward improving performance of database systems, distributed 
database systems, conversion and language aspects of database systems. 

For the many thousands who may not be familiar with the concepts of database 
management systems, a short primer may be appropriate so that the panel dis¬ 
cussions and technical sessions will make some sense. 

What is database management? Roughly, this means the establishment, and 
maintenance of computerized collections of information about an enterprise (man¬ 
ufacturing company, bank, hospital, school or other organizations). This infor- 
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mation, stored in permanent files, contains historical, current and future planning 
information about the enterprise. All this information is collected and preserved 
at considerable expense so that it will be available when needed; to answer a 
question, to assist in making a decision, or to guide someone as to the next task 
to execute. This collection of information has been called the database, as it 
forms the basis or foundation upon which the operations of an enterprise is built. 

Database data represents one of the two kinds of data found in a computerized 
information system. The other kind is message data. The easiest way to under¬ 
stand the difference between the two kinds of data is to ask the question “what 
happens after the data has been processed?” Database data is characterized as 
being put away and remaining static until some process subsequently requests its 
retrieval. Message data, on the other hand, is active. It is being transferred by 
one process because it is relevant to the actions of another process which is 
anticipating its receipt. One or both of these processes may be manual and 
outside of the computer. Thus, the message data and database data complement 
each other and represent the “hurry up” and “wait” of the computerized infor¬ 
mation systems. Database Management is the title given to the art of storing and 
retrieving data. Message Management and Communications Management are the 
titles given to logical and physical aspects of moving data between processes. 
The reader is urged to look at the sessions with titles using the term “commu¬ 
nications” or “networks” or “distributed systems” to find additional material 
on the current state of the data movement technologies. 




Network database evaluation using analytical modeling* 


by TOBY J. TEOREY and LEWIS B. OBERLANDER 

The University of Michigan 

Ann Arbor, Michigan 

INTRODUCTION 

The proliferation of computerized databases and the wide¬ 
spread demand for timely access to data has resulted in the 
need to develop automated techniques for database design. 

In the past, database design was accomplished manually by 
data processing personnel through past experience and trial 
and error. However, the implementation of large integrated 
databases in organizations made it increasingly difficult for 
individual users to do the necessary design. A new approach 
was required to consider the many tradeoffs among user 
objectives and within the complex logical structures required 
to integrate the various types of data. 

The stages of database design have only recently been 
well-defined, 5 and much of that work is still done manually 
with some assorted design aids available at some of the 
stages. One of the more quantifiable stages in the database 
design process is the implementation of a physical database 
from a logical database design. Logical database design re¬ 
sults in the definition of record types and their relationships 
under the constraints of a particular data model, but inde¬ 
pendent of physical structure. The design of the physical 
database must take into account the variations in access 
methods, i.e., the access paths and search mechanisms 
available, as well as physical clustering of records near each 
other within blocks or in a contiguously defined set of 
blocks. Considerably more insight is required in physical 
database design before determining whether or not it can be 
accomplished independently of logical database design. 

Currently there exist some very general physical database 
designers 10,13 which address tradeoffs among major classes 
of storage structure. On the other hand, several evaluators 
of physical databases have also been proposed. Some are 
simulation models and are quite expensive to run. 2,7,9 Others 
are probabilistic and are limited by expected value assump¬ 
tions. 3,11 

This paper described the concepts leading to the devel¬ 
opment of an operational Database Design Evaluator 
(DBDE), which accounts for the more significant parameters 
affecting physical database design performance and yet is 
computationally fast enough for consideration as the central 
module of a physical designer. The DBDE is a software 


package which evaluates Honeywell’s Integrated Data Store 
or IDS. 6 While the model produces “expected value” re¬ 
sults, it goes beyond the usual expected value assumptions 
and considers the effects of the distribution of logical record 
sizes and placement. The DBDE is directly applicable to 
CODASYL database systems. 

The most easily identifiable forerunner to the DBDE was 
the general analytical method of evaluating physical data¬ 
base designs for single record type files proposed by 
Yao. 13,14 The general method decomposed all candidate or¬ 
ganizations and access methods into several basic parame¬ 
ters: the number of index levels, the expected search length 
at each index level and the level of usable data, and the 
fraction of sequential accesses versus pointer (random) ac¬ 
cesses at each level. All applications were assumed to con¬ 
sist of randomly addressed operations such as queries and 
updates. 

The File Design Analyzer 11 extended the basic method to 
consider both sequential and batched processing in addition 
to random processing, and it extended the concept of the 
pointer (random) access parameter to specify the degree of 
randomness required by each configuration. An example of 
this is the implementation of overflow in an indexed se¬ 
quential organization. The pointer to an overflow record 
could refer to another point on the same track, another track 
with the same cylinder, a different cylinder or even a dif¬ 
ferent device. Each specification would imply a different 
expected access time to the overflow area. This extension 
enabled the File Design Analyzer to determine a range of 
performance between “single access” (i.e., a dedicated de¬ 
vice with no seek time, but including an average rotational 
delay between consecutive sequential accesses) and “multi¬ 
access” (i.e., a shared device with the worst case conditions 
of all consecutive sequential accesses becoming random 
accesses in reality). The purpose of this dichotomy was to 
place bounds on I/O service time for the extremes of one 
user and many users. No attempt was made to determine 
the effect on queuing delays with this model or with the 
DBDE. 

The DBDE does, however, represent significant improve¬ 
ments over the previous models in the following areas: 


* Supported by the Defense Communications Agency, Contract Number 
DCA 100-75-C-0064 for the WWMCCS ADP Directorate, Reston, Virginia 
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1. Computation of database overflow probabilities on the 
basis of database growth and distribution of record 
size, and determination of the effect of increasing 
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overflow on the I/O service time required to execute 
various data manipulation operations. 

2. The effect of buffering on I/O service time. 

3. The modeling of “currency” and dependent sequences 
of operations performed on network databases. 

Other analytical models have been confined to storage 
structures for flat files or hierarchical systems, but have not 
considered network databases. The proliferation of COD- 
ASYL DBTG or similar implementations for network da¬ 
tabases have increased the need for network evaluation 
models. 

INTEGRATED DATA STORE (IDS) 

The Integrated Data Store (IDS) is a host language Da¬ 
tabase Management System which interfaces extensively 
with COBOL. It was developed in 1963 by General Electric 
and is now being used on the Honeywell 6000 series and 
other configurations. It is widely regarded as a forerunner 
of the CODASYL DBTG concepts, and is very close to the 
specifications of the 1971 DBTG report. 4 It was developed 
to organize information so as to minimize redundancy and 
duplication of records or data, provide a single database for 
many applications, store and retrieve logically related data 
in cohesive manner, and provide an efficient query capabil¬ 
ity. 6 

The structure of IDS may best be summarized in terms of 
the interrelationships between records and access mecha¬ 
nisms, and chain classification. 

Record classification 

A record may either be a master record or a detail record. 
Masters and details are related to each other by means of 
chains. A chain may contain only one master record. How¬ 
ever, a record may be the master or detail of more than one 
chain, and a record may be a master in one chain and a 
detail in another chain simultaneously. All IDS records of 
a specific record type are fixed length and fixed format. 
Different record types may have different lengths and for¬ 
mats. 

Another form of record classification is by the retrieval 
access mechanism. Three methods are allowed in IDS: 

1. Primary records—retrieved by reference codes which 
point directly to the physical page and line number on 
which the record resides. The page is the basic unit of 
transfer between secondary storage and main storage. 
It is a physical block (with fixed size in subfiles, some¬ 
what analogous to DBTG areas) that may contain many 
record types simultaneously. PAGE RANGE is a fea¬ 
ture of IDS that specifies a set of physically contiguous 
blocks (pages) to contain records of a given type or 
types. 

2. Calculated records—retrieved by calculating a page 
address (via hash code) from a particular key value. A 


sequential search in the page and possibly an overflow 
page yields the desired record. 

3. Secondary records—These are records that can only 
be retrieved by accessing the master records and then 
traversing the chain to which they belong. Often it is 
possible that the immediate master cannot be accessed 
directly. In this event, master records that are on a 
higher level than that of the required immediate master 
need to be accessed. The search is then performed 
from these higher level records down to the immediate 
master in question and then finally to the desired sec¬ 
ondary record. Such higher level masters are known as 
entry points and are accessed via primary record op¬ 
eration, calculated (CALC) record operation or by a 
RETRIEVE master record operation. 

Chain classification 

In IDS a record may belong to any number of chains. If 
retrieval of a record is specified by a particular chain, it is 
retrieved via that chain. If the chain through which the 
record is to be retrieved is not specifically stated, the record 
is retrieved via its prime chain which is declared a priori (in 
the RETRIEVAL VIA CLAUSE). This manner of retrieval 
by means of a chain applies only to the secondary records. 

Finally, a chain may be implemented as any combination 
of the options chain PRIOR (backward pointer) and chain 
MASTER (pointer to the master record). Every chain has 
a NEXT (forward) pointer. 

MODEL PARAMETERS 
DBDE inputs 

Categories of inputs and outputs are summarized in Figure 

1 . 

a. Database Description 

The IDS model requires the definition of master records 
and detail records of chains, the page size and page range 
of subfiles, and the record lengths and page ranges of all 
record types (see Table I for an example). 

b. Workload Specification 

The model identifies applications in terms of specific IDS 
operations called: RETRIEVE, HEAD (i.e., find master 
record), STORE, DELETE, MODIFY, and QUERY. An 
application can be defined by a sequence of related retrieval 
operations, which traverse the network in various ways, 
ending with a specified operation on the desired record(s). 
It is also possible to specify location mode in terms of 
database entry points and type of access method, and the 
PLACE NEAR option. PLACE NEAR allows the cluster¬ 
ing of detail records near master records they are most 
frequently processed with. 
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Figure 1—The data base design evaluator (DUDE) 


c. Hardware Characteristics 

Hardware specifications are, for the most part, limited to 
the timing characteristics and capacity levels of the second¬ 
ary storage subsystem. It is assumed that only one type of 
secondary storage device is used, although the alternative 
types of devices could be evaluated by reinitializing these 
parameters. In IDS, allocation across several devices is pos¬ 
sible. 

Other input parameters which fall loosely under the cat¬ 
egory of hardware are the estimated CPU time required to 
initiate I/O starts and completions, the CPU time for moving 
data within main storage, and the main storage required for 
support software plus user applications. 

One parameter that falls in none of the above categories 
is the specification that the effect of overflow on perform¬ 
ance is desired. The analyst may specify any number of time 
periods that overflow performance data is to be computed, 
with the database growth rate implicitly specified by the 
current database size and the number of record additions 
per time period. All computations in the DBDE are based 
on a unit time period. The analyst may choose the most 
meaningful time unit for his analysis, and merely needs to 
be consistent with that time unit when specifying workload 
parameters and overflow time periods. 


DBDE outputs 

The DBDE outputs reflect an attempt to present a fairly 
complete picture of resource usage and the components of 
response time. Enough performance data is given so the 
analyst can estimate the real cost of the applications eval¬ 
uated. 


a. Main Storage Requirement 

This is a straightforward computation, adding the user 
estimate for object program storage space to the space re¬ 
quired for buffers for each application. 

b. Secondary Storage Requirement 

In IDS this is input explicitly by the user (analyst) in terms 
of subfile specifications for first and last page numbers. 

c. I/O Service Time 

I/O time implies elapsed I/O service time, which affects 
either response time or turnaround time. It differs from 
channel time in that it always includes all the major com¬ 
ponents of I/O service: seek time, rotational delay, and data 
transfer. Bounds on I/O time are provided for the extremes 
of “single access”, and “multi-access” configurations de¬ 
scribed above. 

d. CPU Time 

The DBDE estimates the CPU time required for the da¬ 
tabase functions: overhead to start and complete I/O com¬ 
mands (for physical reads and writes), time for data move¬ 
ment between buffers and user work areas where applicable, 
database search time, and software write verification. The 
user CPU time to process logical records is merely echoed 
from the input data, and is an optional feature. 


MAJOR COMPONENTS 

An algorithm for database sizing and overflow 
computation 

Given a database with N records of varying size, an al¬ 
gorithm to determine its size needs to answer the following 
questions: 

1. How many physical blocks (pages) will be required if 
the database is organized sequentially and the ordering 
results in random placement of records of a given size? 

2. How many record occurrences will typically fit into a 
block? 

The first question is of interest in modeling the record 
distribution or population of sequential and indexed sequen¬ 
tial database organizations, which can be computed by the 
DBDE. The second question is of interest in modeling the 
placement of chains in IDS. 

If all records were of the same size, R words, and the 
physical block size were B words, then the number of rec¬ 
ords that could fit into a block would be 

where [expr]" denotes the “floor function” or “greatest 
integer equal to or smaller than” the actual value of the 
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TABLE I—Input data for test network database 


HARDWARE SECTION 


Secondary storage device 

Honeywell 6060/DSS191 

Average seek time 

30 mi Hi sec 

Adjacent cyl. seek time 

10 mi Hi sec 

Avg. rotational latency 

8.35 mi Hi sec 

Full rotation time 

16.7 mi Hi sec 

Bytes per word 

6 

Transfer rate 

1,074,000 byte/sec. 

Cylinders per disk pack 

404 

Tracks per cylinder 

19 

Bytes per track 

11904 bytes 

Block gap size 

12 bytes 

CPU time for I/O start + end 

1 mi Hi sec 

CPU time for core-core move 

2.75 microsec/byte 


IDS SECTION 
IDS DATA DIVISION 


Number of buffers (Max. is 256) 
Write verify ON? 

Main storage reqmt. for software 
CALC chain retrievals/time unit 
RETRIEVE EACH operations/time unit 


SUBFILE DIVISION 

SUBFILE 1 

SUBFILE 2 

SUBFILE 3 

First page number 

1 

101 

151 

Last page number 

100 

150 

200 

Page size 

320 words 

320 words 

320 words 

Disk number 

1 

2 

3 


50 

NO 

TOOK bytes 
0 
0 
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TABLE I—(Continued) 


RECORD DIVISION 








Max. 100 record types 


Record number 


1 

2 

3 

m 

5 

6 

■ 

8 

Record length (words) 

10 

90 

60 

35 

65 


25 

75 

40 



Retrieval via 


CALC 

CHN.6 

CALC 

CALC 

CHN.2 

CHI.4 

CHN.8 

CHN.10 



No. occurrences 


50 

50 

50 

50 

50 


50 

50 

50 



First page number 


1 

1 

101 

151 

151 

1 

1 

101 



Last page number 


100 

100 

150 

200 

200 

100 

100 

150 



Workload per time 

unit: 













a. Retrieve direct 

5 

5 

5 

5 


5 


5 

5 

5 



b. Retrieve record 

5 

5 

5 

5 


5 


5 

5 

5 



c. Retrieve current 

5 

5 

5 

5 


5 


5 

5 

5 



d. Store 


15 

15 

15 

15 

15 


15 

15 

15 



e. Delete 


10 

10 

10 

10 

10 


10 

10 

10 



f. Modify 


0 

0 

0 

0 


0 


0 

0 

0 



CHAIN DIVISION 









Max. 

100 Chain types 


Chain number 

1 

2 

3 

4 

5 

6 

■ 

8 

9 

10 

Master record 

3 

4 

1 

1 


2 


1 


2 

1 

1 

3 

Detail record 

4 

5 

5 

6 


6 


2 


7 

□i 

8 

8 

Chain order 

FIRST 

FIRST 

FIRST 

FIRST 

FIRST 

FIRST 

FIRST 


FIRST 

FIRST 

Link to next 

YES 

YES 

YES 

YES 


YES 

YES 

YES 

WBm 

YES 

YES 

Link to prior 

NO 

NO 

NO 

NO 


NO 

NO 

NO 

NO 

NO 

NO 

Link to master 

NO 

NO 

NO 

NO 


NO 

NO 

NO 


NO 

NO 

Record selection 
Retrv next/^ 

CURR 

0 

• 

CURR. 

0 

CURR. 

0 

CURR. 

0 

CURR. 

0 

CURR. 

0 

UNIQ. 

0 

CURR. 

0 

CURR. 

0 

CURR. 

0 

Retrv.master/'jl^ 

0 

0 

0 

0 


0 


C 


0 

0 

0 

0 

Head opns./^ 

0 

0 

0 

0 


0 


C 


0 

0 

0 

0 

Place near master 

NO 

YES 

NO 

YES 

NO 

YES 

NO 

YES 

NO 

YES 
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expression. The resulting database size would be 



blocks 


where [expr] + denotes the “ceiling function” or “smallest 
integer equal to or greater than” the actual value of the 
expression. 

The problem becomes significantly more difficult when 
there are a variety of record sizes and not much is known 
about their distribution in the database. The database size 
is extremely sensitive to record placement when there are 
very few record sizes, but a large difference in those sizes. 

An example will help illustrate the possible extreme val¬ 
ues. Suppose there were 1000 records of length 20 words 
and 1000 records of length 90 words and a constant block 
size of 100 words. If the ordering specified all the large 
records must come first and all the small records second, 
the total size of the database would be: 


DATABASE SIZE = 


1000 


noon - + nooi~ 
L 90 J J LL 20 J 


1000 


= 1000 + 200 
= 1200 blocks 


On the other hand, an ordering which alternated large and 
small records would require 2000 blocks. Any other ordering 
of records would produce a database size somewhere be¬ 
tween these two extremes. For very large databases this 
nearly 2-to-l difference in size will greatly affect the esti¬ 
mated performance for many applications, and consequently 
record distribution must be represented accurately in the 
model. 

The algorithm assumes that the size of any record in the 
database is independent of the size of its neighbors. If this 
assumption is not true for sequential or indexed sequential 
organizations, then the database size must be computed 
manually. The IDS model allows dependent record place¬ 
ment to be specified in terms of page ranges. The algorithm 
then enumerates the possible combinations of record sizes 
that would overflow a physical block and require the pop¬ 
ulating of the next block, and it obtains a distribution of 
.unused space in each block. The expected used space in a 
block, called E(used), is easily computed as the difference 
between block size B and the expected value for unused 
space. The total database size is therefore computed as 


DATABASE SIZE 

= [ 2 Record_sizej*Number_of_occurrenceSj/E(used)] + 

ALL 

RECORD 

TYPES; 


The algorithm takes into account parameters such as per¬ 
cent fill and page overhead associated with specific storage 
structures in order to compute realistic values for E(used). 12 

As an example of the type of analysis required, let the 
database consist of 1000 records with an expected record 
size of 200 words and a block size of 320 words. The actual 
distribution of record size is: 

Record Type Record Size Number of 

(Words) Occurrences 


A 

100 

300 

B 

200 

400 

C 

300 

300 


For this simple configuration, a random distribution of rec¬ 
ord types across the database results in the seven possible 
arrangements of records in a block shown in Figure 2 oc¬ 
curring with the probability Pj, i = 1, 2, . . . , 7. From the 



Next record A,8orC B or C C A,B,orC A,B,orC B or C A,B,or C 

type in 
random 
sequence 

Probabil- .027 .063 .09 .12 .12 .28 .30 

ity of 
occurr¬ 
ence, P.j 

Used space 300 200 100 300 300 200 300 

in the block, 

USED. 


Figure 2—Random sequence distribution of record types in blocks 


data in Figure 2 we can compute the expected value of used 
space in each block and then the database size. 


E[used] = 2 Pi * USEDj = 247.7 words 

i=l 

3 

Total data volume = ^ 

3=1 

Record—size j * Number _of_occurrences 


where j is the number of record types 
= 100 * 300 + 200 * 400 + 300 * 300 
= 200,000 words 

Expected number of blocks required 

r 200,000 words , = 808 blocks 


247.7 words/block 


where a record type is defined by its length and number of The same number of fixed size records of 200 words each 
occurrences. would require 1000 blocks. Therefore a significant reduction 
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is possible in this simple case when the record sizes are 
distributed over a larger range. 

The above computation becomes extremely difficult as 
the number of record types becomes large (i.e., 500 or 
more), and even the computer time to exhaustively enu¬ 
merate all cases is prohibitive. The DBDE uses an algorithm 
to compute E(used) that reduces the computational com¬ 
plexity to a feasible range for 50-100 record types, which is 
adequate for most IDS databases. The complete derivation 
of this algorithm is given in Reference 12. 

A natural extension of the database sizing algorithm is to 
consider new records being added to the database and the 
probability that they will overflow the page where they 
would be naturally placed. From this value one can deter¬ 
mine the expected number of overflow records generated by 
a given number of record additions over a particular time 
period. Overflow in IDS significantly increases the expected 
retrieval time for records. 


Buffering 

The original purpose for buffering on second generation 
(uniprogramming) computer systems was to provide a mech¬ 
anism for overlapping CPU and I/O time and improve overall 
performance of each application. Buffering for multipro¬ 
gramming systems still produces an overlap of resources, 
but not necessarily for the same application (job). While job 
A is executing, job B may be inputting into its buffer while 
job C is done with I/O and waiting for the CPU. The exis¬ 
tence of a large number of jobs in the system causes other 
delays which usually subsume the synchronous CPU-I/O 
sequences of a sequential data processing operation. Con¬ 
sequently, the benefits of such buffering received by se¬ 
quential or random database applications in multiprogram¬ 
ming environments are not easily measurable with 
probability models, such as DBDE, but they can be better 
approximated by a queuing model for the whole system. 

The more directly measurable function of buffering in 
multiprogramming systems is that of keeping active data in 
main storage in order to reduce physical I/O operations 
when data is continually referenced. The DBDE considers 
this form of buffering, and it allows the analyst to specify 
up to 256 buffers for IDS applications. Naturally, increasing 
the number of buffers increases the probability that the next 
record you need is already in main storage and consequently 
decreases the expected I/O service time. However, increas¬ 
ing the buffer size too much may result in wasted storage, 
increased cost for storage used, and main storage allocation 
delays that may degrade overall performance. Therefore the 
choice of proper buffer size is critical to an effective IDS 
application. The DBDE implements the buffer concept by 
maintaining buffer currency data based on the most recently 
executed IDS operations and the database definition of 
chains and record types. It also simulates the basic IDS 
buffer replacement algorithm, least-recently-used (LRU). 
Thus, for this part of the IDS model, the DBDE uses a 
hybrid analytical and deterministic simulation technique. 


Currency in a network database 

IDS (as well as DBTG databases) has currency indicators 
which specify the most recently accessed record of a partic¬ 
ular record type or master and detail records for a chain 
type, for example. The DBDE maintains similar information 
to represent the current status of access to the database. 
Consequently the next operation is evaluated according to 
current position and next position desired. The logical struc¬ 
ture diagram of the database is implicitly known from the 
form of the input parameters, and the algorithm consults 
this information to compute I/O time and CPU overhead to 
access the next record. 

EXAMPLE NETWORK DATABASE EVALUATIONS 

The objectives of the test network database evaluations 
were to validate the IDS model with live test data from an 
existing database, to provide insight regarding the sensitivity 
of database performance to the values of storage structure 
and user workload parameters, and to compare the perform¬ 
ance of alternative database designs. 

The conclusions from preliminary validation tests ai * that 
the DBDE is able to describe the major database design 
parameters and their relationships accurately, that the input 
data can be created easily and quickly, and that many mean¬ 
ingful experiments can be conducted at a single interactive 
session with the program. The validation IDS database 
which represented a large equipment inventory system, was 
supplied by a client U.S. government agency. The database 
consisted of 155,000 records of varying length across three 
major record types, and live test data was collected on an 
IDS system interfaced with a Honeywell 6060 GCOS oper¬ 
ating system for an extensive series of retrieval operations. 
The test was conducted during peak hours in a large batch 
multiprogramming environment. Estimates for elapsed time 
for the live test data (54* 10 3 seconds) and the DBDE 
(42.7* 10 3 seconds) differed by 21 percent. The ability of the 
DBDE to estimate total elapsed time was due to the avail¬ 
ability of an extension which implements a central server 
queuing model of the test system and estimates total elapsed 
time based on CPU and I/O requirements derived by the 
IDS model for the test database. Although these preliminary 
results have been encouraging, we believe that the validation 
process should be a continually on-going one as more de¬ 
tailed monitor data is made available. Validation at the in¬ 
dividual IDS operation level is currently not possible in the 
existing environment. 


Example 1 

The first test database chosen for experimentation was 
the support database for DBDE, 1 which provided the op¬ 
portunity to evaluate our own design for DBDE before ap¬ 
plying the technology to client systems. The test database 
schema is illustrated in Figure 3. The database contains eight 
record types and ten chain types. The database is quite small 
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Figure 3—DBDE database schema 


(755 records), but this kept the cost down while allowing a 
great deal of analysis of the model. Other input parameters 
are summarized in Table 1. 

The basic output produced by the program is illustrated 
in Table II. Each operation is considered separately and 
totals are produced for given frequencies of occurrence of 
each operation. Table II illustrates a two-operation appli¬ 
cation that retrieves a record and modifies it. To accomplish 
this task, 2 sub-operations are required for each retrieval 
and 5 sub-operations are necessary for the modify operation. 
This feature of providing detailed output for each operation 
enables the analyst to obtain a better picture of cause and 
effect relationships for overall system performance, and thus 
make better decisions on parameter values to choose. 

Example 2 

The estimates of database I/O requirements for several 
configurations are summarized in Figure 4. The purpose of 
this experiment was to investigate the tradeoff between I/O 
time and secondary storage space for this experiment. It 
consisted of a single master record type (100 occurrences) 
and five detail record type (5000 occurrences each) with 
each contained in a separate chain type. The secondary 


TABLE II—IDS retrieval and update application. Sample output 


APPLICATION: Retrieve and Modify Record 7 





1/0 Service Time (MLSEC) 

System 

CPU Time 

Estimate (MLSEC) 

ACTION 

Record 

Type 

Chain 

Type 

Single 

Access 

Multi- 
Access 

Retrieve as CALC 
record 

1 

- 

10.20 

40.40 

2.77 

Retrieve as chain 
detail record 

7 

8 

.07 

.26 

.02 

Modify and delink 

7 

- 

50.69 

200.70 

8.50 

Retrieve master w/o 
currency table 

2 

7 

10.14 

40.14 

2.75 

Link to chain 7 

7 

7 

20.28 

80.28 

2.00 

Retrieve record 1 
as CALC 

1 

- 

10.20 

40.40 

2.77 

Link to chain 8 

7 

8 

20.28 

80.28 

2.00 

TOTAL 

- 

- 

121.86 

482.46 

_ 

20.81 
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storage space for this configuration was: 



CHAIN 

NEXT 

CHAIN 

NEXT, PRIOR, MASTER 


Master 

Records 

100*(144 bytes) 

100*(164 bytes) 


Detail 

Records 

25000*(128 bytes) 

25000*(136 bytes) 


Total 

3214K bytes 

3416K bytes 


The implementation of pointers for PRIOR and MASTER 
in each record represents an overall increase of slightly over 
6 percent in total storage space. The results illustrated in 
Figure 4 show that the I/O time for most operations is in¬ 
dependent of chain pointer implementations. However, 
there is a slight increase in the STORE and a very significant 
decrease in HEAD (i.e., retrieve master) operations. The 
results for “multi-access” were relatively the same as those 
in Figure 4 but all values were approximately 30 percent 
larger on a point-by-point basis. The increase for STORE is 
due to a greater number of pointers to reset, while the 
decrease for HEAD is due to direct access to the master 
record with a MASTER pointer. Overall, this experiment 
shows that choice of pointer implementation depends on the 
types of applications to be run. Normally, a 6 percent in¬ 
crease in storage space is not significant, therefore, the 
increase in storage space appears to be less important than 
the effect on I/O time, which in turn affects total response 
time. 

Example 3 

We now return to the data structure for Example 1 to 
illustrate the capability of the DBDE to predict degradation 


I/O 

service 
time in 
milliseconds 



Figure 4—I/O service time for IDS operations with two pointer 
implementations (“single access” with dedicated disk) 



Figure 5—I/O service time for the retrieve record 7 operation as a function 
of increasing IDS database size 


in I/O service time as the probability of overflow increases. 
Using the database configuration in Table 1, we assume a 
database growth rate (number of record occurrences of each 
type) of 10 percent per time period for an arbitrary time 
period length. The IDS model derives the overflow proba¬ 
bility at discrete time period intervals for as many intervals 
as the analyst wishes to specify. The assignment of a value 
such as “3 days” for a time period has no effect on the 
model; real time units for time periods are independent of 
the overflow algorithm. 

I/O service time to retrieve a record (type 7) is shown to 
degrade at a nonlinear rate over 20 time periods in Figure 
5. The linear 10 percent database growth rate is included for 
comparison. The purpose of conducting this type of exper¬ 
iment is to compare storage structures, not just for the 
current configuration, but also for future configurations that 
involve more record occurrences and possibly a different 
level of workload. Observation of the results in Figure 5 
could also help determine when and how often the database 
should be reorganized to clear the overflow areas. 

FUTURE DIRECTIONS 

Experience in the development of the DBDE has led to 
the conclusion that considerable insight can be gained into 
the causes and effects of physical database performance. 
Based on this experience it is felt that reasonable extensions 
can be made in three major directions. The first potential 
extension needed is to model the interface between the 
DBMS and the operating system in its many facets: mutual 
exclusion of processes attempting to read and update the 
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same data, queuing delays from mutual exlusion or other 
resource contention, and details of operating system over¬ 
head to facilitate DBMS commands to access data. The 
simulation model of Hulten and Soderlund 7 has achieved 
considerable success in this area. 

Secondly, the evaluation can be extended to databases 
based on hierarchical or other logical models of data because 
they exhibit the same properties as network models: access 
can be described in terms of some combination of direct 
(hashing) or indexing plus a series of sequential or pointer 
steps within a particular level of indexing or at the final level 
of data. The generalization applies to existing search meth¬ 
ods, indexing methods, and overflow techniques, although 
the resulting implementation in the analytical model is more 
arduous in some cases. At the level of detail exemplified by 
the DBDE, each technique must be modeled individually, 
and although models designed by Yao 14 and Severance 10 are 
more general, they lack important detail available in the 
DBDE. 

The third major extension would be to incorporate the 
DBDE into a physical database designer that optimizes user 
specified performance measures as a function of user spec¬ 
ified control parameters. Typically, the control parameters 
would be defined as a subset of the database characteristics 
currently input to the DBDE. Several of these characteris¬ 
tics are fixed (i.e., bound) at the logical database design 
phase, and so the range of physical design control parame¬ 
ters is further narrowed. In IDS one could control subfile 
page size, page range for subfiles and record types, PLACE 
NEAR specifications, RETRIEVAL VIA clauses, and 
pointer options. 
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INTRODUCTION 

A database is a physical representation of information de¬ 
scribing logical entities and relationships of interest to users. 
A database system functions both to maintain the accuracy 
and integrity of the representation as information changes 
over time, and to provide access to subsets of data as re¬ 
quired by users. When choosing storage structures for a 
database, a designer is continually faced with tradeoffs be¬ 
tween data maintainability and data accessibility. 

The design problem is a difficult one. Typically, the num¬ 
ber of design variations is enormous, and the choice of an 
efficient organization is dependent upon numerous problem 
characteristics: 

1. the data to be stored—its sizes, volumes, and volatility; 

2. the data storage environment—its operating costs and 
access characteristics; 

3. the data retrieval applications—their frequencies, 
priorities, response requirements, and data selection 
criteria; and 

4. the system’s design objective—criteria for measuring 
performance, and an associated set of operational de¬ 
sign constraints. 

Precise mathematical formulations of realistic design prob¬ 
lems generally reveal the presence of integer decision vari¬ 
ables, nonlinear constraints and discontinuous objective 
functions. As a result, traditional mathematical optimization 
techniques cannot be applied to database design problems 
in a straightforward way. 

Research aimed at systematizing the database design 
process is supported at a number of American 
universities 1,3 ' 12,21-23 and research laboratories. 14-16 Our in¬ 
vestigation at the University of Minnesota has evolved from 
research carried on first at the University of Michigan 17,18 
and then at Cornell University. 7,9,11 Our approach to data¬ 
base design uses a model composed of (1) parametric de¬ 
scriptions of modular components of a generalized database 
organization, (2) analytic cost equations which evaluate pro¬ 
posed modular database designs, (3) an analyst interface 


which accepts and evaluates arbitrarily selected designs, and 
(4) search procedures which automatically generate and 
compare thousands of alternative organizations. 

The purpose of this paper is to describe a methodology 
devised to permit our automatic design procedures to search 
a large set of alternative record access paths efficiently, 
while selecting the best combination for a given design prob¬ 
lem. 


THE MODELING CONTEXT 

Figure 1 provides an overview of the modular, generalized 
database organization assumed in our research. While three 
major modules are distinguished, our concern here is with 
the design of Module A—the file structures which define 
access paths to subsets of database records. Results of proj¬ 
ects to design search techniques (Module B) 4,19,20 and record 
structures (Module C) 2,7,10 are reported elsewhere. 

We assume each retrieval request, input in Figure 1, is 
known and given by: 

a. a relative frequency or priority of retrieval, 

b. record selection criteria, based upon field values stored 
within data records, 

c. attribute projection criteria defining fields to be re¬ 
trieved from selected records, and 

d. ordering criteria for the presentation of retrieved data. 

To appreciate the combinatorially explosive nature of the 
access path selection problem, consider an employee data¬ 
base and the three retrieval requests described by Table I.* 
Any one of these requests can be satisfied by scanning all 
employee records in a database which is arbitrarily organ¬ 
ized. Records of interest would be selected on the basis of 
DEPARTMENT and SKILL values and sorted as necessary 


* Projection criteria are essential for the design of efficient record structures 
(Figure 1, Module C), but are ignored here. 
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GENERALIZED DATABASE ORGANIZATION 


DATABASE UPDATES 


RETRIEVAL REQUESTS 


DATA EXTRACTS 


Figure 1—A component-level overview of the database design model 


for presentation to the user. Since each subset is likely to 
be small, a number of alternative file structures might be 
used to organize these record collections for faster re¬ 
trieval. 20 For example, one might connect the records of all 
engineers together in a list file in order by DEPARTMENT 
(Request 1), or construct an inverted list which points di¬ 
rectly to all members of department A in order by NAME 
(Request 2). The “engineers in department A” (Request 3) 
could be retrieved via a scan and sort from either of these 
files, or a separate access path could be established for this 
request as well. 

Each of the alternatives has a number of structural vari¬ 
ations which trade record accessibility for database main¬ 
tainability. Depending upon its expected rate of growth, a 
file may vary in its initial loading density or in its use of 
chained overflow areas to accommondate new records. List 
structures may or may not use back-pointers to facilitate 
record deletions. The type of pointers used within inverted 
files may vary depending upon the expected database vol¬ 
atility. Every ordered structure may use any of a number of 
order-value search techniques to speed insertion and dele¬ 
tion operations. 

In general, consider a design problem with N retrieval 
requests, each of which on the average could be retrieved 
from any of K x distinct implementations of K 2 different file 
types. The problem of access path selection, then, is equiv¬ 
alent to choosing from among {K x K 2 ) N alternative database 
designs. Typical values for K t K 2 and N, in the range 20 to 
50, make an exhaustive search of the solution space com¬ 
putationally infeasible. 


TABLE I.— A Simplified Description of Three Data Extracts 


Request 

Number 

Frequency 
(per month) 

Selection 

Criteria 

Order 

Required 

1 

20 

SKILL= ENGINEER’ 

DEPARTMENT 

2 

5 

DEPARTMENT=‘A’ 

NAME 

3 

10 

DEPARTMENT = ‘A’ 
and SKILL= 
‘ENGINEER’ 

PASS NUMBER 



OPTIMAL ACCESS 
PATH 

COMBINATION 


• Data to be Stored 

• Data Storage Environment 

• Data Retrieval Applications 

• System Design Objectives 



• Structural Definition 

• Implementation Parameters 

/fr 7' 51- 


Figure 2—Overview of the access-path design procedure 


THE PROCEDURE FOR DESIGNING ACCESS PATHS 

A three-stage design process has been devised to select 
an efficient combination of access paths on the basis of the 
system’s total cost. The procedure takes as input a quanti¬ 
tative description of the data to be stored, the data storage 
environment, and the data retrieval applications. Total cost 
is measured as the sum of storage, retrieval and maintenance 
costs. A detailed description of the analytic models used to 
calculate the total cost of a proposed design is well beyond 
the scope of this paper (Duhne 5 provides a thorough discus¬ 
sion). An understanding of the optimization procedure is, 
nevertheless, easily conveyed without substantial detail. 
Consider the basic design procedure as illustrated by Figure 
2 . 

The first step in the procedure generates a large set of 
potentially useful access paths. Only a few will actually be 
selected in the final design. To insure the computational 
tractability of the optimization procedure, it is desirable to 
restrict the number of access paths considered. On the other 
hand, to insure the optimality of the selected design, no 
reasonable access path should be excluded. The set of paths 
is generated from the analysis of retrieval requests. 5 Each 
access path in the set is described generically in terms of its 
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Cost Curves For Successively 



Figure 3—Typical access-path cost function 


record membership criterion, its record ordering, and its 
basic file type (e.g., sequential file, inverted file, etc.). 

The second step shown in Figure 2 determines an optimal 
implementation for each potential access path at every fea¬ 
sible* level of retrieval activity. As was mentioned earlier, 
a variety of implementations trade record accessibility for 
data maintainability. For each level of retrieval intensity, a 
search procedure developed by Severance 17 and refined by 
Duhne 5 is used to locate an optimal implementation. Intui¬ 
tion correctly tells us that, as the frequency of file access 
increases, selected implementations will successively reduce 
the marginal cost for retrieving the file. 

For each potentially useful access path, a cost function is 
output from Step 2. As illustrated by Figure 3, the function 
defines the minimal total cost (for storage, maintenance, and 
retrieval) at every level of retrieval intensity. The imple¬ 
mentations which yield these minimal costs are stored for 
later recall. Duhne 5 shows that total cost is a continuous, 
non-decreasing, concave function of retrieval intensity. This 
is an intuitively reasonable property, and it is essential in 
the final setp of the design procedure. 

The third step in Figure 2 assigns retrieval activities to 
access paths in a manner which minimizes total system cost. 
A branch-and-bound procedure is used—the reader is di¬ 
rected to Garfinkle and Nemhauser 8 for a thorough discus¬ 
sion of this optimization technique. Generally, branch-and- 
bound procedures implicitly enumerate all possible solutions 
to a problem by means of a search tree built from nodes and 
labeled branches. The root is associated with the complete 
solution space. Each other node corresponds to a restricted 
class of solutions derived by imposing solution restrictions 
which label branches on the path from the root to that node. 



RETRIEVALS PER TIME PERIOD 

Figure 4—Cost function bounds in the interval (R1,R2) 


A search proceeds as follows: Starting with the root, a 
series of nodes in the tree are “visited.” With each visit, 
upper and lower bounds on the cost of solutions reachable 
from the current node are computed; a least upper bound 
for the entire tree and its corresponding node are continually 
recorded. On the basis of the cost calculations at the current 
node, a branch is chosen for exploration. The search pro¬ 
ceeds down the tree until a visited node is “fathomed” and 
no branches from that node are explored. 

A node is fathomed either (1) when the upper and lower 
bounds for the node are sufficiently close that all solutions 
governed by the node are effectively equivalent, or (2) when 
the lower bound of the node is greater than the current least 
upper bound for the tree so that no solution governed by 
the node can be optimal. 

The search algorithm “backtracks” from a fathomed node 
to its parent node and proceeds to search other unexplored 
branches in the tree. When none are left, the search is 
complete. The (any) solution defined by the node with the 
current least upper bound is selected as optimal. 

This general procedure is applied to our specific problem 
by traversing a search tree in which each node is defined by 
a collection of retrieval activity intervals {[/G*, jR 2,]}” =1 , one 
for each of n potential access paths, i. Every solution gov¬ 
erned by a node (i.e., reachable from it) must be composed 
of an optimal implementation for each access path given 
these activity intervals.* So that every possible solution is 
considered, the root of the search tree is defined by 
{[0, RMAXi ]}&!, where RMAX\ is the sum of all retrieval 
activities which could potentially be satisfied via access path 
i. 

An upper and lower bound on the total cost of each access 
path is computed at each node visited. Since the actual 
function Ci(R) is known to be concave, a lower bound in the 


* The maximum retrieval intensity for each access path is calculated as the _ 

cumulative frequency of all user retrieval requests which can use that path. * If RU=R2i=0, then, optimally, the access path is not implemented. 
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interval [JRlj, R2 t ] (as illustrated by Figure 4) is given by 
the line Ll t passing through Ci(Rli) and Ci(R2i). An upper 
bound is given by the intersecting lines L2 t and L3* (passing 
through points Cj(/?li) and C t (R2i) with slopes dCi(Rli)/ 
dR and dCi(R2i)/dR respectively.) 

Using the lower bound approximation, extracts are as¬ 
signed to the access paths from which they can be retrieved 
at least cost. Cost is calculated as the sumt of: 

(1) the cost of access path retrieval which is taken to be 
the slope of LI*, and 

(2) the cost of extract sorting, which is a function of the 
size and retrieval criteria of the extract, and calculated 
using statistics which describe the sort utility of the 
host system. 13 

Duhne 5 has shown that the total cost of any solution 
reachable from a node is bounded by the sums of upper and 
lower bounds for the individual access paths. If the current 
node is fathomed by these bounds, the search returns to its 
parent node. If the node is not fathomed, two new branches 
are generated by partitioning the retrieval interval of a se¬ 
lected access path into two parts. The interval to be divided 
is chosen so that the difference between the upper and lower 
bound (at the point of cumulative retrieval assignments) is 
greatest. The partition boundary is set at the point in this 
interval where the difference between its current bounds is 
greatest.* The search proceeds as described for the general 
algorithm and terminates when an optimal collection of ac¬ 
cess paths has been determined. 

The design procedure has been encoded as a FORTRAN 
program and run on a CDC CYBER 74 at the University of 
Minnesota. 6 It has been applied successfully to a number of 
realistic design problems using less than a minute of CPU 
time. The cost estimates generated by our analytic models 
have been verified via comparisons with actual costs of 
design implementations. Model accuracy has ranged from 
one to six percent. 5 

SUMMARY 

The application of mathematical optimization techniques to 
the difficult problem of database access path design has been 
described. The design process is divided into three steps. 
Generic descriptions of potentially useful access paths are 
first developed from user retrieval specifications. An optimal 
implementation for each access path at every feasible level 
of retrieval intensity is then determined. For each path a 
function describing minimum total cost for every retrieval 
intensity is constructed. Using the fact that these functions 
are continuous, non-decreasing, and concave, a branch-and- 
bound procedure is used to locate access path implementa- 


t A slightly more complex calculation can be used to model the possibility of 
file intersection and union during extract retrieval; see Reference 5. 

* As an exception, the first partition boundary on an interval (0, RMAX t ) is 
always set at RM1N { defined by the minimum retrieval frequency over all 
requests which access path i can satisfy. 


tions which can be used at minimal cost for the retrieval of 
user extracts. The result is an efficient database design. 

The design procedure has been applied to realistic prob¬ 
lems and executes in a reasonable time. The accuracy of our 
analytic cost models has been verified through implemen¬ 
tations of selected designs. Details of these models have 
been purposely suppressed in an effort to highlight the struc¬ 
ture of the optimization procedure which we believe is gen¬ 
erally applicable to the problem of access path selection. 
Since the procedure basically requires only that a cost func¬ 
tion (with reasonable properties) be constructed for each 
access path to be considered, we feel that the algorithm can 
be readily adapted to the modeling work of others. 
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Programming languages for relational data base systems* 
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Berkeley, California 


INTRODUCTION 

There has been a remarkable growth in the use of data base 
management systems over the past decade. This has resulted 
from the recognition that many practical applications, pre¬ 
viously implemented by special purpose programs, can be 
developed more easily through the use of a general data 
base facility. Here, we will only be concerned with relational 
data base systems. The advantages of such systems have 
been discussed at length in the literature. 7,9,10 

In modern data base management systems, users typically 
interact with data bases via a query language. 4,6,8,20,41 These 
languages, which are often interactive, allow creation and 
deletion of data bases as well as retrieval from, and updating 
of, existing data bases. Because query languages have not 
provided a complete programming environment, most of 
these languages have been coupled to existing programming 
languages. Application programs consist of statements in 
the programming language intermixed with statements in the 
query language to access the data base system when re¬ 
quired. 

Previously investigated approaches to providing such ac¬ 
cess include the definition of subroutines to execute data 
base functions and the embedding of data base constructs 
into an existing language, using a preprocessor to translate 
these constructs into run-time calls on a data base system 
(e.g., EQUEL, 2 and the embedding of SEQUEL in PL/1 6 ). 
Because the primary focus of this work was to provide data 
base access from a programming language, little attention 
was paid to the programming environment resulting from 
the combination of the two languages. 

While a given programming language and a given query 
language may each be satisfactory in isolation, their com¬ 
bination may be less than satisfactory because neither one 
was designed for this more general setting. In this paper we 
discuss techniques for improving the programming environ¬ 
ment for data base applications. Our hypothesis is that a 
superior environment can be realized by incorporating the 
data base operations as part of the programming language 
itself. In the second section we outline some limitations 
associated with previously proposed environments. These 


* This research was supported in part by U.S. Army Research Office Grant 
DAAG-29-76-G-0245, and by the University of California under a Faculty 
Research Development Grant. 


stem primarily from the fact that the programming language 
is totally unaware of the data base. In the third section we 
illustrate some of the benefits provided by an integrated 
approach through the application of current techniques from 
programming language research. 


CURRENT APPROACHES 

Most current approaches to providing data base access 
from a programming language are constrained by the fact 
that the language system has no knowledge about the data 
base. We divide our discussion of the impact of this con¬ 
straint into five areas: data base interface, data base oper¬ 
ations, type system, compilation, and abstraction. 

Data base interface 

As mentioned previously, the two primary approaches to 
providing data base access have been to provide subroutines 
for data base functions and to embed data base constructs 
through the use of a preprocessor. 

The subroutine approach suffers from the limitations in¬ 
herent in any use of subroutines to extend the semantic 
space of a language; programs become essentially a series 
of calls which are often unreadable because of the many 
(bookkeeping) arguments which must be passed to the sub¬ 
routine. No syntactic aids are provided to make the use of 
these arguments understandable. 

The preprocessor approach is somewhat better in that an 
actual query language is used. Program statements and 
query language statements can be freely intermixed. In most 
systems the preprocessor does not have to parse the entire 
language because query language statements are tagged with 
some distinguished symbol. The preprocessor removes the 
query language statements, replacing them with calls upon 
the data base subsystem, before the program is passed to 
the language translator. The programmer transacts with a 
readable version of the program and only the language trans¬ 
lator need be concerned with the less readable version. 

Unfortunately, the preprocessor approach is limited as 
well. Since the query language is not part of the program¬ 
ming language, communication between the programming 
language and the data base system can become awkward. 
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For example, it is often necessary to use implicitly defined 
global variables to communicate the status of queries. In 
some contexts, program variables which are to be made 
accessible to the data base system must be preceded by 
“special symbols” so that they may be recognized by the 
preprocessor. In some preprocessors arbitrary constraints 
are imposed as to which arguments to a data base operation 
can be variables and which must be constants. In other 
implementations, even though the language being extended 
is strongly typed, the same level of checking is not extended 
to the data base objects and their operations. Detailed ex¬ 
amples illustrating these problems are given in a previous 
paper. 35 

Although preprocessing the entire language would remove 
some of these limitations, this approach is still constrained 
by the fact that (practically speaking) no amount of prepro¬ 
cessing can extend the language to incorporate relations with 
the same amount of symmetry, type checking and protection 
available for built-in objects. 

Data base operations 

In considering the query language as part of the program¬ 
ming language one finds that the notation and orientation of 
the two are quite different. Commonly used programming 
languages are procedural, with built-in facilities for iteration, 
manipulation of complex data structures, and procedural 
abstraction. On the other hand, most query languages are 
non-procedural. In the context of a procedurally oriented 
programming language these non-procedural operations 
seem out of place. To provide a consistent programming 
environment the objects manipulated by the data base sys¬ 
tem and their attendant operations and control structures 
should be related to the objects, operations and control 
structures of the language itself. For example, a join oper¬ 
ation may be thought of as a nested iteration over relations 
yet the operation “join” usually appears in a form com¬ 
pletely unrelated to the iteration operations in the program¬ 
ming language. We do not advocate that higher level oper¬ 
ators such as joins be excluded from a language. Instead, 
we suggest that they be defined as extensions to the language 
in terms of existing primitives. 


Type system 

Because the programming language has no knowledge of 
the data base system, the type systems of the two are usually 
distinct. Data base objects cannot be manipulated in the 
programming language with the same ease and flexibility as 
language objects. For example, records, as used in a rela¬ 
tional data base system, often are not treated as composite 
structures in the language environment (e.g., EQUEL or 
SEQUEL). This means that values must be copied individ¬ 
ually from data base records into program variables before 
the values can be used. Thus, a record from a data base 
cannot be an argument to a procedure nor an operand to an 
otherwise meaningful operation (e.g., record selection or the 


right-hand-side of an assignment). The same situation holds 
for relations. For example, relations may not be passed as 
parameters to functions. In addition, it is impossible to op¬ 
erate on relation types, say, to obtain the count of the 
number of columns in the relation, or to obtain the domain 
of a column as a type. The existence of two type systems 
may require conversions to be performed on objects as they 
flow between the two systems. One type system controlling 
data access whether in main memory or secondary storage 
would be more efficient because it removes the need for 
such conversions. 

Compilation 

In programming languages, there is often a tradeoff be¬ 
tween program flexibility and run-time efficiency. Most data 
base systems are committed to one extreme or the other 
depending upon whether data base operations are inter¬ 
preted (less efficient but more flexible,* e.g., INGRES) or 
compiled (more efficient but less flexible, e.g., SEQUEL). 
When a language is committed to one extreme the resulting 
design decisions make the language awkward to use, or less 
efficient, when the application requirements are not matched 
to that commitment. For example, consider the treatment 
of relation names in SEQUEL. In order for a query to 
appear explicitly in the text, all relation names must be 
constant (to allow compilation to be performed). If dynam¬ 
ically varying names are desired, then a string for the query 
text must be constructed and the relation name inserted into 
it at run-time. On the other hand, INGRES allows dynami¬ 
cally varying relation names but cannot utilize the fact that 
a relation name is constant in a given instance. A better 
programming environment would allow for a continuous 
spectrum ranging from dynamically varying requests with 
less efficient execution to fixed requests with more efficient 
execution so that the system can provide the best possible 
solution for the requirements of the problem to be solved. 

Abstraction 

The concept of data abstraction has received much atten¬ 
tion in the programming language community. Recently, 
there has been interest in the use of data abstraction with 
respect to data base systems. 18 Many data base systems 
support abstraction related facilities such as views, integrity 
constraints and triggers. 3 * 12,19 ’ 33 For example, a view pro¬ 
vides a form of abstraction in that the user can construct 
queries for relations which do not actually exist but are 
materialized from a number of relations. If the organization 
of the relations is changed, the view can be maintained by 
redefining it in terms of this new organization. In addition, 
some authors have proposed mechanisms for specifying 
higher level operations on data bases (such as “hire” and 
“fire”). 18 Although some systems offer these abstraction 


* Requests can be changed at run-time (e.g., in EQUEL relation and domain 
names can change dynamically). 
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facilities, they have not yet integrated them with their coun¬ 
terparts in data abstraction from programming languages.* 


AN INTEGRATED APPROACH 

We will now consider the integration of data base objects 
and operations into a programming language. We will not 
attempt to give a complete language proposal here. Instead, 
we shall illustrate some of the benefits of this approach. Our 
discussion is divided into four areas: type system, data base 
operations, compilation, and abstraction. 


Type system 

An extremely important aspect of the language is the 
extension of the type system to provide access to data base 
relations residing on secondary storage. While a complete 
discussion of the type system is beyond the scope of this 
paper, we will discuss a few aspects of the type system to 
demonstrate the usefulness of an integrated approach. We 
have been strongly influenced in our thinking about types 
by the Mesa language. 15 

In introducing data types for relations in the programming 
language we will exploit the obvious correspondence be¬ 
tween relations and collections of records in programming 
languages, namely, a relation may be viewed as an unor¬ 
dered collection of records (tuples in data base terminology), 
where the collection may not contain duplicate tuples. For 
example, 

Employee: type—relation of 

name: string(20), 
dept: integer, 
salary: integer, 
manager: string(20), 
jobtitle:string(15) 
end 

defines Employee as a relation type, where “relation of’ 
means “unordered collection with no duplicates of . . .”. 

A relation type is itself an object upon which operations 
may be performed to obtain access to types contained within 
it. Thus, 

declare e: record(Employee) 

declares e as a variable whose type is identical to the un¬ 
derlying record type of the relation, and 

declare deptno: Employee.dept 

declares deptno to be a variable whose type is identical to 
the dept column of the relation, i.e. integer. 

By allowing the types contained within the relation type 


* Recent work in this direction has been reported. 28,32 


to be extracted in this manner, duplication of these types in 
the program is avoided. Thus, a degree of modularity is 
achieved. In addition, data integrity is enhanced because the 
only operations that can be performed on the values in the 
data base are those allowed on the given type. Because there 
is only one representation for a value of a given type, 
whether in the data base or in the program environment, 
and because the set of types provided in the language is the 
same as those provided in the data base system, no conver¬ 
sion is needed when data is moved from one environment 
to the other. 

Another important feature of the type system is the ability 
to specify constant or dynamically varying relation and do¬ 
main names. For example, the following program fragment 
declares Rel to be a relation bound to the existing data base 
relation EMP. 

begin 

declare Rel: Employee=relation(‘EMP’); 

end\ 

If the relation were not constant, e.g., if its name were 
entered at a terminal, the program fragment would be 

begin 

declare Rel .Employee, 

Rname: string(15); 

write(“What is the relation name?”) 

read(Rname); 

Rel: =relation(Rname) 

end', 

where relation is a function which takes a string argument 
and returns a relation as result (or returns undefined if no 
such relation exists). In either case the type of the relation 
is specified so that operations on Rel can be type-checked, 
and the program code in the block can be the same. The 
only change is in the way the relation is bound to Rel. 
Because the programmer has explicit control over the con¬ 
version of the input string into an object of type relation, 
the existence of the named relation can be tested and an 
appropriate action taken if it does not exist. This cannot be 
done easily in any existing language. 

The last aspect of the type system to be discussed is type 
extensibility. Several researchers have mentioned that a 
type extension facility would be valuable so that types other 
than the typical primitives (e.g., integers, reals, and char¬ 
acter strings) can be placed into a data base record. The 
proposed language will allow arbitrary fixed length struc¬ 
tured objects, except those including pointers or relations, 
to be a component of a data base record. Providing this 
feature is not particularly difficult if the data base system 
supports a typing mechanism which can be extended to 
include generic functions.* 39 


* Generic functions have multiple definitions for an operator where the par¬ 
ticular definition invoked at a call is determined by the arguments (e.g., + 
has several definitions, including definitions for integer, real, complex or 
mixed arguments). 
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Data base operations 

There are several ways that data base operations can be 
made more convenient, consistent, and more closely related 
to existing constructs in programming languages. Our inter¬ 
est in this research was motivated by the difficulty one 
author of this paper had expressing complicated queries in 
SEQUEL. This led to the design of a procedural query 
language, described in detail elsewhere. 26 In the following 
discussion, an overview of the query constructs is pre¬ 
sented. 

Suppose that we wish to print the names of all employees 
whose manager is Kathy Fallon. This is written as 

for x in Rel st x.manager=‘Kathy Fallon’ do write(x.name); 

end 

and read “for each record x in the relation Rel, such that 
x’s manager is ‘Kathy Fallon’, print x’s name.”* The vari¬ 
able x is known only in the scope of this statement and is 
bound to each record in the relation satisfying the qualifi¬ 
cation. The qualification clause (st) is optional. 

We believe that this construct is cleaner than those used 
in many existing data base languages. The binding of lan¬ 
guage environment variables to data base objects is more 
consistent and the control structure is more natural, say, 
than requiring explicit testing of a variable shared by the 
data base system and the program environment to determine 
if another record exists. 3 

A query which requires the join of two relations may be 
written as a nested for construct. For example, a list of all 
employees who earn more than their managers could be 
obtained by writing 

for emp in Rel do 

for mgr in Rel st emp.manager=mgr.name do 
if emp.sal>mgr.sal then write(emp.name) fi 

end 

end 

Suppose one wanted to construct a temporary relation 
having employee records for all people working in depart¬ 
ment 50. This could be accomplished by using the for con¬ 
struct to append qualifying tuples to a (temporary) relation 
variable. 

Alternatively, the for statement can be used in its short 
form 

begin 

declare Reltemp: employee; 

RelTemp:=[«// | Rel:dept=50] 

end 

which says “construct a relation composed of those records 
in Rel such that dept is 50 and assign it to RelTemp.” The 
construct on the right hand side of the assignment operation 
is an example of a relation constructor. The keyword all 


* The use of a for statement to sequence through a collection of data objects 
has been proposed before by programming language designers 13,17,21,30 ' 31 but 
is absent from existing data base languages. 


indicates that all fields of the record are selected. A relation 
constructor causes a copy of each record to be made. If 
only the name and salary fields were needed, the expression 
would be 

[name,salary | Rel:dept=50] 

Of course, to assign this value to RelTemp, the type speci¬ 
fication for RelTemp must be changed. 

The value returned by a relation constructor may contain 
duplicate records. Such objects are called bags, 27 or multi¬ 
sets. 23 If the value is assigned to a relation variable, then 
duplicates are removed. However, there are situations in 
which the value is useful with duplicates retained, e.g., when 
a function such as average is used as in AVG([salary|rel]). 
To explicitly remove duplicates from the value, two vertical 
bars (“I”) are used. A relation constructor may be used 
directly in a for statement without having been previously 
bound to a relation variable. This is demonstrated by the 
next example which intermixes query iteration, relation con¬ 
structors, and functions which take relations as arguments 
to print a list of all departments and the average salary of 
employees in that department. 

for x in [dept || Rel] do 

write(x.dept,AVG([salary | Rel: dept=x. dept])); 

end 

This is read “for x in the set of distinct departments from 
Rel, print x and the average salary for those employees in 
department x’ ’. 

The iterative construct is also used to update individual 
records as shown in the next example in which all employees 
are given a 10 percent raise. 

for x in Rel do 

x.salary: =x.salary * 1.1; 

end 

As in most relational systems, the relation is not actually 
changed until all iterations are completed because the stor¬ 
age structure for the relation may require that records be 
physically moved when a field is changed, in which case a 
record could be updated more than once. One solution, used 
in some set theoretic languages, is to make a temporary 
copy of the set over which the iteration variable ranges. 13 
This would not be practical in this environment because of 
the size of relations and the time needed to make a copy. 
Alternatively, the problem may be solved by keeping all 
changes in a separate file until the end of all iterations. 34 
This also solves the problem of keeping a data base consis¬ 
tent after certain kinds of crashes. 

Compilation 

The linguistic benefits provided by an integrated environ¬ 
ment will be of little utility in practice unless reasonably 
efficient code can be produced. Here, we discuss a number 
of compilation techniques which can be utilized in order to 
ensure that programs in the language have an efficient re¬ 
alization. The topics discussed are: variable binding time, 
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partial evaluation of procedures at compile-time, and pro¬ 
gram optimization. 

Variable binding time 

Binding time is the time at which a variable is bound to 
an entity, e.g., a specific relation to a variable of type re¬ 
lation. Whether this binding occurs at compile-time or run¬ 
time influences both run-time efficiency and program flexi¬ 
bility. Figure 1 shows a graph of binding time of variables 
in data base queries versus the degree to which these queries 
are compiled for two relational data base systems. 

Compiling is more efficient in execution time and may be 
more efficient in execution space, while late binding allows 
more flexibility. Notice that these two systems make explicit 
commitments, to flexibility (INGRES) or to execution effi¬ 
ciency (SEQUEL). By using compilation and optimization 
techniques developed for implementing programming lan¬ 
guages, it is possible to develop a system that operates in 
the grey region of the graph. When enough parts of a lan¬ 
guage construct are fixed (e.g., the relation name is constant) 
code as efficient as possible can be compiled. 22 Otherwise, 
less efficient code is compiled. This means that the tradeoff 
between efficiency and flexibility can be controlled by the 
applications programmer. Furthermore, it means that only 
those programs that need more flexibility must pay for it in 
terms of efficiency. Of course, in those cases where code is 
compiled to access a specific relation, programs may have 
to be recompiled if the definition of the relation changes. 

Partial evaluation 

A second compilation technique which can be used is the 
partial evaluation of procedures at compile-time (procedure 
closure). In SEQUEL, all relation names are viewed as 
either constants in the program text (in which case compi¬ 
lation is performed) or as strings “read in” at run-time (in 
which case no compilation is performed). With this organi¬ 



zation it is impossible to pass a relation name to a procedure 
and to compile queries issued from the procedures which 
use this relation name. However, with global flow analysis 
techniques 16 combined with procedure closure techniques, 37 
a programmer may be given a number of choices as to how 
such a procedure call should be compiled. These include: 

1. compile one procedure body using only the type infor¬ 
mation about the relation, 

2. compile distinct procedure bodies for each of the dif¬ 
ferent (constant) relations passed to the procedure 
using type information as well as specific information 
about the relation, or 

3. expand certain calls “in-line” so that no call is actually 
made. 

Providing a programmer with a facility such as this gives 
one considerable control over the tradeoff between execu¬ 
tion time and space. 

Optimization 

The use of an integrated language opens up many possi¬ 
bilities for optimization. The goal is to reduce the number 
of calls on the data base system (and subsequent disk ac¬ 
cesses) by combining many queries into one, removing quer¬ 
ies from loops, eliminating temporary relations, and so on. 
This can be achieved through the application of traditional 
programming language optimization techniques (e.g., code 
motion, common subexpression elimination, and loop fu¬ 
sion). 1 However, the potential payoff from such optimiza¬ 
tions is much greater in data base applications than in typical 
programs because the removal of a disk access represents 
a savings many orders of magnitude greater than the removal 
of, for example, an assignment statement. 

.Beyond traditional types of optimizations, possibilities 
exist for more complex optimizations using techniques sim¬ 
ilar to those used in experimental programming systems 
which have program verification capabilities. 38 In this case, 
semantic information available in an abstraction containing 
relations (e.g., integrity constraints) and information about 
the physical organization of the data can be used to perform 
sophisticated optimizations. 

Abstraction 

Data abstraction facilities have been found useful in de¬ 
veloping programs that involve complex data structures. 11,24,40 
It seems reasonable to consider extending such facilities to 
handle relational objects and operations. Another reason for 
exploring such a facility is because it may provide a struc¬ 
turing mechanism for current data base concepts such as 
views, integrity constraints, and triggers 3,12,19,33 which are all 
related to abstraction. Currently these facilities appear in 
data base systems in an unstructured fashion which often 
makes it difficult to determine how they interact and relate 
to one another. A better approach would be to make them 
all aspects of a single abstraction mechanism. 
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A view permits a programmer to conceive of a data base 
in terms of virtual relations. This allows fields, or complete 
relations, to be operated on even though they might not 
exist physically. This is similar to the notion of uniform 
reference that is being explored in programming languages. 14 
Another use of views is to prevent a programmer from seeing 
certain columns of a relation. Again this is similar to a 
programming language concept (see the private attribute in 
Mesa 15 ). A problem with using views is that it is not always 
clear how a value to be stored to a virtual field should be 
decomposed and stored into the fields that are physically 
present. Another problem is that it is not always clear what 
action should be taken if a value is to be assigned to a field 
which would cause the record to fall out of the view. Some 
data base researchers propose to solve these problems by 
establishing defaults where unambiguous alternatives are 
possible and disallowing the operations otherwise. An alter¬ 
native solution, which we are currently investigating, is to 
acknowledge the difficulty of establishing defaults and in¬ 
stead provide convenient tools for allowing the implementer 
of the abstraction to specify the semantics explicitly. 

Triggers are data base operations which are invoked au¬ 
tomatically when certain primitive operations are per¬ 
formed, e.g. when a tuple is deleted from the employee 
relation the count of employees in the removed employee’s 
department is decremented by one. In the context of a data 
abstraction facility it would seem more reasonable to asso¬ 
ciate this operation with the abstract operation “fire” than 
with the deletion operation itself. 

Integrity constraints are assertions which characterize 
what it means for the data base to be in a correct and 
consistent state with respect to the semantic meaning of the 
data. For example, a constraint might be that all the names 
in the data base must be a subset of those in the employee 
relation. 19 Currently, such constraint information is used in 
an enforcement sense, i.e., a constraint is checked after a 
data base operation (or a group of operations) are performed. 
Often these checks are very expensive. It may be possible 
to use this information as assertions to be proved to hold 
about the set of abstract operations which are permitted on 
the data base. For those constraints which cannot be proved 
to always hold it may be possible to automatically determine 
additional information which should be retained in order to 
simplify enforcement. 

CONCLUSIONS 

We have attempted to show that an integrated approach to 
providing data base access from a programming language 
can yield a better programming environment for data base 
applications than previously available. We are currently in 
the process of designing a language with this philosophy for 
use with the INGRES 20 system. Other research groups are 
also working on similar problems. 5,25,29,36 We believe that 
this research represents a significant step towards a better 
understanding of the relationship between programming lan¬ 
guages and data base systems. 
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INTRODUCTION 

Over the past few years, technological advances in database 
management have introduced many new research and de¬ 
velopment problems. The problem of “application migra¬ 
tion” of a database management system is one of them. It 
can be grossly divided into two closely related subproblems: 
the problem of database translation and the problem of ap¬ 
plication program conversion due to database changes. 
There are many reasons for database reorganization and 
restructuring. Some of these are (1) the change of the use 
environment of a DBMS, (2) the upgrading of hardware and 
software facilities, (3) the change from one database system 
to another, (4) the installation of a new DBMS, (5) the 
modification of an existing system for efficiency purposes, 
and (6) the change of database semantics. The database 
translation problem has been tackled with some success by 
the works reported in References 7-9, 13-16, 18, 20-24, 29. 

Some database changes, both structural and semantic 
ones, may require that the set of application programs writ¬ 
ten for the old database be modified or converted to account 
for the changes either for semantic reasons or for efficiency 
purposes. A well-designed DBMS, which is data independ¬ 
ent, should normally not be affected by the changes made 
on the physical structure and access strategy because these 
types of database changes can be handled simply by chang¬ 
ing the mappings from the external model to the conceptual 
model and from the conceptual model to the internal model. 6 
However, few existing database systems are truly data in¬ 
dependent. Physical placement of data and access strategies 
are often carried into the application programs. 3 ’ 4,12 Fur¬ 
thermore, some semantic changes of database will inevitably 
cause changes in both the schema and the subschema which 
the data sublanguage directly interacts. 

Very often, the sublanguage statements as well as the host 
language statements need to be modified. As with the area 
of database translation, application program conversion 
could feasibly be implemented by the user or system pro¬ 
grammer, however, the task is such an enormous one that 
the personnel’s time is not effectively optimized. Therefore, 
a need for some type of program conversion system is def¬ 
initely warranted. 


* This research is supported by NSF Grant MCS 76-10075. 


The work on application program conversion is still in its 
infancy. In the paper by Su, 25 a brief review of some existing 
works related to this problem was given. An attempt was 
made to analyze the problem and to identify the tasks which 
need to be undertaken to obtain some kind of automatic or 
semi-automatic aid for program conversion. In the same 
work, a general model of program conversion as it is related 
to database translation was presented. It was proposed that 
two stages of conversion are involved with the program 
conversion problem in the DBMS environment: 

(1) Sublanguage query conversion whereby query analy¬ 
sis and modification occur due to schema and sub¬ 
schema changes, and 

(2) Host language program conversion whereby the trac¬ 
ing of program logic, the identification of variable re¬ 
lations and program-subprogram structure represent 
some of the needed analyses. 

Since the problem of program conversion is such a com¬ 
plicated one, we have decided to narrow our initial investi¬ 
gation to a restricted area in the first stage of program 
conversion, i.e., the sublanguage query conversion. In order 
to get a better handle on the problem, the following assump¬ 
tions and restrictions are made: (1) The data model used in 
the DBMS is the relational model. 5 (2) The subschema is 
identical to the schema of the database. (This assumption 
does not nullify the problem investigated here because of 
the types of database changes considered here.) (3) The 
DEFINE language 10 is used as the data description language. 
(4) The CONVERT language 19 is used as the data mapping 
language. The CONVERT language is designed for process¬ 
ing and converting hierarchical trees called “forms.” It is 
borrowed here for relational processing because relations 
are special cases of general “forms.” (5) The relational 
sublanguage SEQUEL 1,2 is used to formulate the source and 
target queries. Only the basic patterns of the language are 
considered. 

The objectives of this investigation are as follows: (1) To, 
study the effects of all the CONVERT operators, (2) to 
design the algorithms for collecting data from the input 
source to construct the data tables to form the translation 
dictionary, (3) to study the techniques for analyzing a limited 
yet general SEQUEL query patterns, and (4) to design a 
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translation algorithm in detail (i.e., to the level of algorithm 
description which is one step short of the actual coding) to 
support the analysis and translation of queries. 

This paper is a short version of a technical report printed 
in May, 1977. 26 Related to this work, two independent works 
on program conversion (Su and Liu 27 and Housel 11 ) take a 
more general approach to the program conversion problem. 
Both works advocate the use of an abstract representation 
of program semantics and the semantics of data translation 
operators. 

In the subsequent section, we shall first describe several 
types of database changes and illustrate the problems of 
sublanguage query conversion with examples which involve 
the use of CONVERT operators and SEQUEL’S basic map¬ 
pings. A translation process is then detailed in the third 
section and an example of query conversion is provided in 
the fourth section to clarify and illustrate the translation 
process step-by-step. A conclusion and the future work are 
given in the last section. 


SOME ORGANIZATIONAL AND SEMANTIC 

CHANGES OF A DATABASE 

We shall describe in this section some organizational and 
semantic changes and illustrate how these changes can be 
brought about by the CONVERT operators and their effects 
on the SEQUEL queries. This description and illustration 
should give the reader a feel of the sublanguage conversion 
problem we are dealing with and of the CONVERT and 
SEQUEL languages used in this study. The data changes 
described here constitute only a small subset of possible 
changes and the description of the languages is not complete. 
The interested reader should refer to the original papers on 
the language. 1,2,19 

In the following presentation we shall use the following 
relations for illustration purposes. 

EMP (E#, ENAME, SEX, MGR) 

PROJECT (P#, PTITLE, BUDGET, DEPARTMENT) 

E-P ( E#, P# ) 

A. A Large Relation is Split into Several Relations 

This type of organizational change may occur in a DBMS 
for data access efficiency and data security reasons. For 
example, the PROJECT relation may be split into several 
small relations, one for each of the departments handling 
the projects. This may need to be done because it was 
observed that most of the application programs access de¬ 
partmental projects separately and the use of different se¬ 
curity constraints are required for different departments’ 
projects. To perform such a database translation, the follow¬ 
ing CONVERT statements can be used. Here, we assume 
that there are three departments only. 

PROJ-Dl<—SELECT (P#, PTITLE, BUDGET FROM 
PROJECT: 

PROJECT.DEPARTMENT=ACCOUNT¬ 
ING) 


PROJ-D2*—SELECT (P#, PTITLE, BUDGET FROM 
PROJECT: 

PROJECT. DEPARTMENT=SALES) 
PROJ-D3<—SELECT (P#, PTITLE, BUDGET FROM 
PROJECT: 

PROJECT.DEPARTMENT=PURCHASING) 

Each SELECT statement selects the tuples satisfying the 
DEPARTMENT qualification and creates a new relation 
with a different name. Note that the DEPARTMENT infor¬ 
mation now becomes implicitly specified by the new relation 
names. 

The effects of this type of database changes are many. A 
SEQUEL query for accessing the tuples of the original PROJ¬ 
ECT relation such as 

SELECT * 

FROM PROJECT 

WHERE BUDGET>50K 

would have to be converted into the following structure. 


SELECT 

* 

FROM 

PROJ-D1 

WHERE 

BUDGET>50K 

U 


SELECT 

* 

FROM 

PROJ-D2 

WHERE 

BUDGET>50K 

U 


SELECT 

* 

FROM 

PROJ-D3 

WHERE 

BUDGET>50K 


The retrieved result is still different from the original query 
because the department information is not explicitly speci¬ 
fied in the new database. The DEPARTMENT field and 
value need to be added into the records retrieved by each 
SEQUEL basic mapping (the SELECT-FROM-WHERE 
block). This can only be done if the conversion system 
stored the DEPARTMENT information of PROJ-D1, PROJ- 
D2 and PROJ-D3 files during the analysis of the CONVERT 
operators. Queries making use of the DEPARTMENT in¬ 
formation or retrieving the information will have to be spe¬ 
cially handled. 

B. Several Relations are Combined into a Single Relation 

This type of database reorganization is necessary in order 
to eliminate the unnecessary duplications and to facilitate 
the access of related data scattered in several relations. This 
is often required when changing a conventional file system 
into a database system. 

To reverse the operation given in (A), the following CON¬ 
VERT statements can be used. 

CONCAT (DEPARTMENT: ACCOUNTING ONTO 
PROJ-D1 AT BUDGET) 

CONCAT (DEPARTMENT: SALES ONTO PROJ-D2 
AT BUDGET) 

CONCAT (DEPARTMENT: PURCHASING ONTO 
PROJ-D3 AT BUDGET) 

PROJECT^-MERGE (PROJ-D1, PROJ-D2, PROJ-D3) 
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The CONCAT operator expands the relations by inserting 
the department data to the right of the BUDGET field. It 
makes the DEPARTMENT data explicit in each project 
relation. Here, the arguments, DEPARTMENT: AC¬ 
COUNTING, DEPARTMENT: SALES and DEPART¬ 
MENT: PURCHASING, are used in place of relation names 
which are required in the original CONVERT syntax. The 
effect of this type of change to the SEQUEL queries is that 
references made to several relations should be changed so 
that reference can now be made to the same relation. This 
generally causes a compression of the number of basic map¬ 
pings to a single mapping such as the reverse operation of 
the example shown in (A). 

C. Changes of the mathematical mappings between/ 
among Entities 

A database can be said to contain data about some entities 
and their associations. Several mathematical mappings be¬ 
tween/among entities have been identified: (1) one-to-one 
association between two entities or within one entity, (2) 
one-to-many association between two entities or within one 
entity, (3) many-to-many associations between two entities 
or within one entity and (4) many-to-many associations 
among more than two entities. Each type of association can 
be changed to any other types due to rule or policy change 
of an enterprise using the database. For example, the rule 
which allows many employees to participate in a project and 
an employee to participate in many projects (many-to-many 
associations between EMP and PROJECT entities) can be 


changed so that one employee can only participate in one 
project but one project can be participated in by several em¬ 
ployees (thus, a one-to-many association between PROJ¬ 
ECT and EMP entities). To represent the new semantics, 
the relations EMP (E#, ENAME, SEX, MGR, P#) and 
PROJECT (P#> PTITLE, BUDGET, DEPARTMENT) 
would be sufficient. If the database contents have been 
changed by data deletion so that one employee participates 
in only one project and it was further assumed that the data 
of employees not participating in any project are to be de¬ 
leted, the CONVERT operator, GRAFT, can be used for 
this database change. GRAFT provides a means of combin¬ 
ing two or more relations into one when specific conditions 
are satisfied. For example, the following statement 

NEW<-GRAFT (E-P ONTO EMP: E-P. E# = 

EMP.E#) 

would create a new relation, NEW (E#, ENAME, SEX, 
MGR, P#), which contains the data of each employee plus 
the project he participates in. If this relation is used to 
replace the EMP and E-P relations in the target database, 
the program translator needs to know at least three things: 
(1) the data of those employees who have not participated 
in the projects have been deleted from the database, (2) the 
EMP relation has been expanded to include P# data and has 
been renamed, and (3) the many-to-many association has 
been changed to a one-to-many association. 

A SEQUEL query which retrieves the project numbers of 
all projects participated by SMITH from the source database 
will have to be modified as follows: 


SELECT 

P# 

SELECT 

P# 

FROM 

E-P 

-* FROM 

NEW 

WHERE 

E# IN 

WHERE 

ENAME=‘SMITH’ 


SELECT 

E# 



FROM 

EMP 



WHERE 

ENAME = ‘SMITH’ 



The nested mappings on the left are compressed and modi¬ 
fied to account for the database changes as shown in the 
new query on the right. In this example, the translation 
analyst will have to be informed of the possible loss of data 
due to the deletion caused by the database change. If the 
facts of data deletion have been recorded, the program con¬ 
version system can tell the analyst precisely what data are 
missing by running the new query against the new database. 

D. Adding or Deleting Entities and/or Associations 

This type of database change requires the modification of 
schema and subschema and often requires changes to the 
application programs. It reflects the dynamic nature of a 
database to meet the new requirements of migrating appli¬ 
cations. Adding new data generally will not affect the ap¬ 
plication program in a well-designed DBMS. Nevertheless, 
it can affect the storage operations (delete, insert and up¬ 
date) of some existing systems. For example, in DBTG’s 


network model, if a new record type is added into a database 
as the AUTOMATIC member of an owner-coupled set 
whose owner was in the database, a deletion statement in¬ 
volving the owner in an old application program may have 
to be modified so that the new record occurrences will not 
be mistakenly deleted because of their AUTOMATIC mem¬ 
bership specification. Deleting data (entity and association 
data) from database will always entail changes to the sub¬ 
language statements as well as the host language program 
itself. Any reference to the deleted data will have to be 
taken care of by the conversion system. The translation 
analyst needs to be informed of the deletion and references 
to the deleted data can be taken out of the program if agree¬ 
able to the analyst. However, this kind of program modifi¬ 
cation may cause a chain of reactions to other program 
statements since they may make use of the deleted data or 
those data which depend on the deleted data. Furthermore, 
some deletions may render the sublanguage statements 
meaningless. For example, a set of sublanguage statements 
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may originally be designed to process data about two entities 
through their association and the association of these two 
entities might have been deleted. The statements would no 
longer be meaningful. In this situation, the statements should 
be taken out and the translation analyst informed of the 
program changes made. 

The foregoing discussion represents a sample of a large 
number of situations in which changes in the subschema and 
thus the application programs are either required or desira¬ 
ble. It also illustrates the types of changes needed to be 
made in the sublanguage queries due to database changes. 
In the examples given, we consider only the effects of a 
single database conversion operator in isolation. In reality, 
the problem is much more complex than what has been 
presented since a set of CONVERT operators might have 
been applied to the source database. In this situation, the 
program conversion system has to keep track of all the 
possible effects on each domain and each relation of the 
source database. It also has to keep track of the conditions 
under which each relation in the target database is created 
and how it is related to the source relations. Sizable data 
need to be collected from the description of source and 
target databases and from the database conversion operators 
applied. 


A SEMI-AUTOMATIC ANALYSIS PROCESS FOR 

SUBLANGUAGE QUERY CONVERSION 

The entire analysis process is divided into six stages as 
illustrated in Figure 1. 

A. Stage 1—Translation Dictionary Development 

The contents of the translation dictionary represent infor¬ 
mation extracted from the source and target database de¬ 
scriptions and from the data mapping operators original ap¬ 
plied to generate the target database. The DEFINE data 
descriptions and the CONVERT statements represent the 
two sources from which the information in the dictionary is 
constructed. The information is useful in the conversion of 
all source database queries. It guides the formal translation 
algorithm (Stage 5) in the construction of new queries which 
will account for the database changes. The translation dic¬ 
tionary contains seven tables. 


(1) TABLE NAME RE ASSIGNMENTS (TNR): This 
table is used primarily to determine if any source database 
table names have been changed during the data conversion 
process. It has the following format: TNR(SOURCE_ 
TABLE_NAME, TARGET_TABLE_NAME). Changes 
in database table names occur mainly in the CONVERT 
language ASSIGNMENT statement, however implied 
changes are also possible in just about all of the CONVERT 
language statements as shown in the following examples. 
The source tables renamed are underlined. 

TABLEO <— EMP(EMP#,DEPT,STATUS) 

SLSPR2(DEPT# .TOTS ALES) <— SALES( DEPT. 

TOTSLS) 

TABLE1<—SELECT(DEPT,NAME FROM DPTTBL : 
DPTTBL.D# =SUP.D#) 

TABLE2«-SLICE(E#,EMPNAME,SALARY FROM 
PERSONNEL ) 

TABLE3*-GRAFT(INVNTY ONTO PTS : 
PTS.P#=INVNTY.P#) 

TABLE4<—CQNCAT( TBL10,TBL20 ONTO TBL30 AT 
S#) 

TABLE5*-MERGE( SERVICE1,SERVICE2 ) 

TABLE6<—SORTt TSBLONE BY S#,P#) 

TABLE7*-ELIM-DUP( TABLETWO ) 

TABLE8<—CASE( T ABLE 15 .SEX=‘Male’,‘Female’) 
(SELECT(E#, ‘ 1 ’, FROM TABLE 15) 
SELECT(E# ,‘0’, FROM TABLE 15)) 

The contents of this table is needed mostly in the reassigning 
process of the FROM clause of a SEQUEL language basic 
mapping. The SEQUEL language FROM clause refers to 
the database table from which the query in question will 
derive its results. 

(2) DOMAIN NAME REASSIGNMENT (DNR): This 

table records the changes on the source domain names. The 
data is used primarily during Stage 4 to develop the query 
pretranslation reference tables which will be described later. 
The table has the following format: DNR 
(SOURCE_DOM AIN_N AME, T ARGET_DOM AIN_ 

NAME). Changes in the database domain names occur 
mainly in the CONVERT language ASSIGNMENT state¬ 
ment, however implied changes are also possible in just 
about all of the CONVERT language statements. Some ex¬ 
amples are given below. The respective source domain 
names are underlined. 


DATABL(STAFF#,PART) <-DATATLB( EMP# ,ITEM ) 

EXMPTBL <-TABLE20(S# RENAME TO SUPPLIER, P# RENAME TO PART 

TABLE25 (DPMT,IDNET) < -SELECT(DEPT,NAME FROM DEPTTBL) 

TABLE26 (BOOK# ,CATLG) <-SLICE(SERLNM, INDEX FROM LIBRYTB) 

TABLE27 (EMP#,SEX)<-CASE (EMPTBL.SEX='‘MALE’, ‘FEMALE’) 

(SELECT®#, ‘1’, FROM EMPTBL), 

SELECT(E#, ‘O’, FROM EMPTBL)) 


The contents of this table is needed mostly in the reassigning 
process of the SELECT, WHERE, AND, and OR 
CLAUSES of a SEQUEL language basic mapping. All of 
these clauses of the SEQUEL basic mapping contain data¬ 


base domain name references which relate to the database 
table present in the FROM CLAUSE of the very same basic 
mapping. 

(3) TABLE DOMAIN USAGE (TDU): This dictionary 
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Figure 1—Stages of query analysis and conversion 


table has the dual purpose of specifying all of the domains 
which a database table contains and all of the database tables 
which contain a certain domain (Figure 2). 

This example shows database table TAB LEI as contain¬ 
ing domains EMP#, MGR and SALARY; PERSONNEL as 
containing domains EMP# and MGR: and, PRODUCT as 


containing domains EMP# and SALARY. This dictionary 
table helps in the query conversion process by making sure 
that all references to database tables and domains in the 
target database are legitimate. It also helps in the reassigning 
of domain names and table names of SEQUEL basic map¬ 
ping which have had extensive name reference changes. 

(4) TABLE CONVERSION INDICATORS (TCI): This 
table specifies the CONVERT operators which have been 
applied on the relations during data conversion. The data is 
useful for determining the order of execution of the formal 
algorithms in stage 5 of the analysis process. The structure 
of this table is illustrated in Figure 3. 

(5) CASE ASSIGNMENT DEFINITIONS (CAD): The 
CONVERT CASE ASSIGNMENT statement allows for 
varied operations to occur over different instances of a da¬ 
tabase table. In essence, it may be desirable to code a value 
or values of a given domain of a database table in a different 
manner for usage in the target database. Therefore, these 
new values need to be stored in the CAD table so that when 
the source domain values are referred during the query con¬ 
version process, the new values may be inserted in their 
place. For example, the following CONVERT statement 
recodes the sex field of the Personnel table (male=l, fe¬ 
male =0) and changes the domain name from SEX to SEX- 
CODE and the table name PERSONNEL to TABLE 1, 


TABLE 1 (EMP#, AGE,SEX-CASE(PERSONNEL.SEX=‘MALE’,‘FEMALE’) 

CODE,RACE)<-(SELECT(EMP#,AGE,T,RACE FROM PERSONNEL), 
SELECT(EMP#,AGE,‘O’ ,RACE FROM PERSONNEL)) 


The above statement causes two entries in the CAD table 
shown in Figure 4. 

The CAD table is accessed by the formal query conversion 
algorithms only after all source database table and domain 
names have been reassigned to their corresponding target 
database references. 

(6) GRAFT CONVERT INFORMATION (GCI): This 
table is related to the CONVERT GRAFT operation. Entries 
in the GCI table simply show which database table was 
grafted onto which table during data conversion. The infor¬ 


mation present in this table contributes greatly to the re¬ 
structuring process of source database queries. In general, 
source queries using database tables which have been 
grafted together may be able to be restructured into a much 
simpler query. 

(7) SELECT OCCURRENCE CONDITIONS (SOC): 
This table holds information concerning the conversion of 
source tables which were operated upon by the SELECT 
operator. For example, the following two SELECT opera¬ 
tions will produce the entries shown in Figure 5. 


TDU 



TABLE1 

PERSONNEL 

PRODUCT 

EMP// 

1 

1 

1 

MGR 

1 

1 

0 

SALARY 

1 

0 

1 


Figure 2—Table for domain usage 
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TCI 





CAD 



ASSIGNMENT 

SLICE 

SELECT 

GRAFT 

... 

CASE ASSIGNMENT 


TARGET 

TABLE NAME 

TARGET 
DOMAIN NAME 

SOURCE 

DOMAIN VALUE 

TARGET 

DOMAIN VALUE 

















TABLE1 

SEX-CODE 

MALE 

1 

EMP 

SALES 

1 

0 

1 

0 

.1 


0 


SEX-CODE 

FEMALE 

0 

TABLE1 

0 

1 

0 







TABLE1 

0 

0 

0 

0 


1 



Figure 4 —Table for case assignment 



Figure 3—Table for conversion indicators 


SELECT(S# ,DES,SNAME FROM 
SLSPR: SLSPR.S# >10) 

SELECT(P#,PNAME,SUPL FROM PTS :PTS.P#=50) 

The information in the SOC table is used to determine if 
a query which makes reference to relations that have been 
operated upon by the SELECT operator will produce any 
result. 

B. Stage 2—Source Database Query Input 

This stage deals with the analysis of the source query to 
determine if the query is one of the appropriate structures 


capable of being operated upon by the formal conversion 
algorithm; and, to verify the labeling of the SEQUEL basic 
mappings present within the source query since the elemen¬ 
tary unit of conversion by the formal translation algorithm 
is the basic mapping. Because of the degree of structural 
flexibility within the SEQUEL language and the original 
concern to narrow down the problem to a manageable size, 
it was decided to curtail the number of acceptable query 
structures. The acceptable SEQUEL structural patterns are 
shown below. It is felt that the structures selected are gen¬ 
eralized enough to allow for the processing of a wide range 
of source queries. 


ACCEPTABLE SEQUEL QUERY PATTERNS 

TYPE I : BASIC MAPPING WITH NO POSSIBLE NESTING REFERENCES: 
SELECT “Domain Name(s)” 

FROM “Table Name” 

WHERE “Conditional Phrase” 

AND “Conditional Phrase” 

OR “Conditional Phrase” 


TYPE II : BASIC MAPPING WITH POSSIBLE OUTER AND/OR INNER REFERENCE MAPPINGS: 
[POSSIBLE OUTER 
MAPPING- 
TYPE II STRUCTURE] = 

SELECT ‘‘Domain Name(s)” 

FROM “Table Name” 

WHERE [POSSIBLE INNER MAPPING— 

AND TYPE II STRUCTURE OR 

OR CONDITIONAL PHRASE] 


TYPE III : BASIC MAPPING WITH CONNECTING INTERSECTION AND/OR UNION OPERATORS: 


SELECT 

“Domain Name(s)” 

FROM 

“Table Name” 

WHERE 

[POSSIBLE INNER MAPPING- 

AND 

TYPE II STRUCTURE OR 

OR 

CONDITIONAL PHRASE] 

u,n 

SELECT 

“Domain Name(s)” 

FROM 

“Table Name” 

WHERE 

[POSSIBLE INNER MAPPING- 

AND 

TYPE II STRUCTURE OR 

OR 

CONDITIONAL PHRASE] 


.further segments are possible 

NOTE: in all of the structural types presented above, the AND and OR clauses are optional. 
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For ease of description and because the formal translation 
algorithms need to be specific in addressing the basic parts 
of a query, it is necessary to develop some terminology 
associated with the contents and structure of a SEQUEL 
query. We will refer to a basic SEQUEL mapping as a 
“segment.” Therefore, depending upon the nesting factor 
of a query structure, a particular basic mapping can be 


defined within a query by its level and segment numbers. 
There can be as many levels as appropriately needed for an 
application. The level within a query changes only when an 
outer or inner mapping occurs in the query itself. Different 
basic mappings used in one level are noted as different 
segments. For example, the query presented below would 
be labeled as follows: 


[LEVEL 1 

SELECT 

SALES,TAX 


SEGMENT 1] 

_FROM 

WHERE 

TABLEONE 

DEPT= 


[LEVEL 2 


SELECT 

DEPT 

SEGMENT 1] 


_FROM 

WHERE 

U 

SELECT 

STORETB1 

MGR =‘ ALAN’ 

[LEVEL 2 


DEPT 

SEGMENT 2] 

AND 

_FROM 

WHERE 
SALESMAN = 

STORETB2 

LOCTYPE=2 

[LEVEL 2 


SELECT 

SALESMAN 

SEGMENT 3] 


_FROM 

WHERE 

TABLETWO 
SALARY >11000 


The information content of each segment (basic mapping) 
within each and every level of a query is coded and saved 
so that mapping relationships and information may be avail¬ 
able for the formal algorithm. The following information is 
derived for every segment within a given level (Figure 6): 
In order to label the information provided above in its level 
and segment context, the following syntax will be used: 

[ABBREVIATED REFERENCE NAME, L,S] 


Now, for a full understanding of the meaning of the query 
labeling terminology, an example will be presented. Note 
that since some of the label references provided are optional 
in the SEQUEL query syntax, the value in such a case 
where there is an applicable information will be ‘null.’ Say 
the query in question was the following: 


SELECT EMP#,CODE 
FROM TABLEONE 
WHERE DEPT-5 
AND SALES+COMM< 1000 

AND MGR= ALLEN’ 


U 


SELECT 

EMP# 

FROM 

TABLETWO 

WHERE 

E# = 

SELECT 

FROM 

WHERE 

AND 

OR 

OR 


E# 

TABLETHREE 
SUM(SALES) = 10,000 
DEPT+CODE=5 
SALARY 
+ COUNT(T#)=20 
MARK=1 


soc 


SOURCE TABLE NAME 

SOURCE DOMAIN NAME 

SPECIAL 

CONDITION 

VALUE 

OPERATOR 

SLSPR 

S # 

10 

> 

PTS 

P if 

50 

= 


Figure 5—Table for selection conditions 


INFORMATION DESCRIPTION 

ABBREVIATED REFERENCE NAME 

LEVEL NUMBER 

L 

SEGMENT NUMBER 

S 

OUTER MAPPING REFERENCING 
CLAUSE DOMAIN 

OMRCD 

SELECT CLAUSE DOMAIN (S) 

SELDOM(N) 

FROM CLAUSE TABLE 

FROMTBL 

WHERE CLAUSE DOMAIN (S) 

WHRDOM(N) 

WHERE CLAUSE ARGUMENT 

WHRARG 

AND CLAUSE (S) DOMAIN (S) 

ANDDOM(C,N) 

AND CLAUSE (S) ARGUMENT 

AUDARG(C) 

OR CLAUSE (S) DOMAIN (S) 

ORDOM(C,N) 

OR CLAUSE (S) ARGUMENT 

ORARG(C) 

INTERSECTION-UNION 

CONNECTION 

INTUNC 


Figure 6—Information contents for each segment 
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Then the following labels represent some examples of infor¬ 
mation which could be derived from the example query: 


[SELDOM(l),l,l] 

= 

‘EMP#’ 

[SELDOM(2),l,l] 

= 

‘CODE’ 

[FROMTBL,l,l] 

= 

‘TABLEONE’ 

[WHRDOM(l), 1,1] 

' = 

DEPT’ 

[WHRARG,1,1] 

= 

‘5’ 

[ANDDOM(l, 1), 1,1] 

= 

‘SALES’ 

[ANDDOM(l ,2), 1,1] 

= 

‘COMM’ 

[ANDDOM(2,1), 1,1] 

= 

MGR’ 

[ANDARG(2),1,I] 

= 

ALLEN’ 

[OMRCD,l,l] 

= 

‘null’ 

[INTUNC,1,1] 

= 

‘U’ 

[SELDOM(l),l,2] 

= 

‘EMP#’ 

[FROMTBL,l,2] 

= 

‘TABLETWO’ 

[OMRCD,l,l] 

= 

‘E#’ 

[WHRDOM(l),l,2] 

= 

‘E#’ 

[FROMTBL,2,l] 

= 

‘TABLETHREE’ 

[WHRDOM(l),2,l] 

= 

‘SALES’ 

[ORDOM(l,2),2,l] 

= 

‘T#’ 

[ORARG(2),2,l] 

= 

‘1’ 

C. Stage 3 — Query Translation Preparation 

Stage 3 of analysis determines which translation proce- 

dures need to be used 

in 

converting each and every basic 


mapping within the query. There are four separate proce¬ 
dures designed to carry out the translation task. The order 
and number of procedures to be used in converting a basic 
mapping depends totally upon the source database table 
being referred to in the FROM clause. Using the TCI table, 
it is possible to search for the source database table to 
determine which CONVERT operations were performed. 
The selection of procedures and their order of execution 
depend on the CONVERT operators applied on the source 
database table as illustrated by the flowchart in Figure 7. 
Thus, if TABLE 1 of a segment were operated upon by the 
CONVERT operators SORT and CASE ASSIGNMENT, 
PROCEDURE 1 and PROCEDURE3 would be called to 
translate the segment. 


D. Stage 4—Query Pre-Translation Reference Table De¬ 
velopment 

During this stage, a set of reference tables is set up 
based on the analysis of a source database query. For each 
basic mapping in a query, the following number of tables are 
established. Their contents are described below. 


NOTE* 

All of the decision 
blocks reference whether 
or not the database 
table of the source 
query basic napping 
has been acted upon by 
the specified CONVERT 
operator. 


START 



PROCEDURE 1 



PERFORM 
+ PROCEDURE 3 


STOP 


Figure 7—Program selection and execution sequence 


REFTB2.n 

This is a set of reference tables (n=0,l,2, . . .) each of 
which is set up for an AND clause of the basic mapping. 
The information stored in these reference tables concerns 
the domain or domains used in the AND clauses and the 
target database tables in which they are contained. 

REFTB3 

The contents of this reference table are derived from an 
analysis of REFTB1 and REFTB2.n tables. It is primarily 
used to determine if there is any target database table which 
contains all of the domains present within the WHERE and 
AND clauses of the basic mapping. 

REFTB4.n 

This set of tables (n=0,l, . . .) has the similar contents 
as the REFTB2.n except that the tables relate to the do- 
main(s) in the OR clauses of the basic mapping. 


REFTB1 

This reference table contains information about the do¬ 
main^) used in the WHERE clause of a basic mapping. 
Each tuple of REFTB1 contains a WHERE clause domain 
(or corresponding domain name reassignment) and the target 
database table in which the domain can be found. This 
information can be extracted from the TDU Dictionary 
Table. 


REFTB5 

The contents of this reference table are derived from an 
analysis of the target database tables which are present in 
REFTB3 and REFTB.n. It is used to determine if there is 
any target database which contains all of the domains pres¬ 
ent within the WHERE, AND and OR clauses of the basic 
mapping. 
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REFTB6 THROUGH REFTB19 

These reference tables hold information concerning the 
domain(s) of the SELECT clause of the basic mapping being 
analyzed. It is assumed that a maximum of 14 domains can 
appear in a SELECT clause. Each table contains the name 
of a domain, all its domain reassignment names and the 
target database tables in which they can be found. 

REFTB20 

The contents of this table are generated by an analysis 
performed over REFTB6 through REFTB19. It lists the 
names of all target database tables in which all the domains 
used in the SELECT clause can be found. 

Q.C.T. 

This is a special reference table whose initials represent 
“Query Conversion Table”. It is used directly by the trans¬ 
lation algorithm to restructure the source database query. 
Its contents are developed through an analysis of REFTB5 
and REFTB20. Query changes due to domain and table 
name reassignments are noted in this table. 

The above set of tables are different from the Dictionary 
Tables generated in Stage 1. The dictionary tables hold in¬ 
formation concerning the entire source database and the 
data conversion information; whereas, the reference tables 
contain data related to one source query. Examples of the 
reference tables are given in the next section where the 
entire translation process is illustrated by an example. 

E. Stage 5 — Formal Translation Algorithm 

Stage 5 uses the data established in the dictionary and 
reference tables to convert each of the basic mappings in 
the source query. The basic mappings are processed se¬ 
quentially from top to bottom (i.e., the segment with the 
lower level number first). As mentioned earlier, there are 
four main procedures used for query translation. These pro¬ 
cedures have been written in PL/1-like statements which 
describe the algorithms of conversion and are given in a 
master’s thesis by Reynolds. 17 PROCEDURE2 is designed 
for taking care of the effect of applying the GRAFT operator. 
PROCEDURE 3 and 4 are concerned with CASE ASSIGN¬ 
MENT and SELECT operations respectively. PROCE¬ 
DURE 1 is a general procedure which handles the conversion 
of basic SEQUEL mappings to account for database changes 
introduced by other CONVERT operators. 

A sizable number of subroutines and functions have been 
defined for accessing data from the dictionary and reference 
tables, for testing the data conditions specified in the basic 
mappings of the source query and for generating the basic 
mappings of the target query. These routines and functions 
are described in the original report. 26 They are not given 
here because of their length. 

F. Stage 6—New Target Database Query 

This stage is concerned with the formation of the new 
target database query. The labeling techniques and inner 


workings of the translation procedures make the final con¬ 
struction of the new target query rather simple to derive. 
There are only two distinct ways in which a query’s input 
structure may be modified: expanding a basic mapping into 
two mappings and contracting two basic mappings into one. 
These two basic structural changes are handled by RE¬ 
STRICTS and RESTRICT5 routines. 

RESTRICT3 is concerned with query transformations 
caused by changes in data access paths. When a query 
structure conversion of this type occurs, the nesting factor 
or total number of levels present within the query increases. 
While this expansion is taking place, the function also read¬ 
justs the labeling criteria of all basic mappings within the 
source database query to compensate for the transformation. 
All reference tables are also reassigned so that they are 
associated with the correct basic mappings. There is no need 
to develop reference table information for the newly created 
basic mapping levels. RESTRICT3 ensures the correctness 
of these particular basic mappings. 

RESTRICT5 is concerned with query transformations 
caused by the use of the GRAFT operator on database 
tables during the data conversion process. In a structural 
conversion of this kind, the nesting factor or total number 
of levels present within a query decreases. RESTRICT5, 
like RESTRICT3 also readjusts the labeling criteria of all 
basic mappings within the source database query to account 
for this structural contraction. All reference tables are either 
reassigned or redeveloped so that proper association is re¬ 
covered. 

Thus, with all structural conversions being taken care of 
by RESTRICT3 and RESTRICT5, the same basic mapping 
processing sequence developed during Stage 2, modified by 
any added and/or deleted basic mappings, can be used to 
construct the new target query. This stage is performed after 
all basic mappings of a source query have been processed 
by Stage 5. 


AN EXAMPLE ON THE SUBLANGUAGE QUERY 

CONVERSION 

A query conversion example presented within this section 
is given in order that the reader may obtain a clearer un¬ 
derstanding of the analysis process and the translation al¬ 
gorithm. More examples of different degrees of complexity 
can be found in a Master’s thesis by Reynolds 17 where the 
degree of complexity of the examples is varied depending 
upon which CONVERT operators were used on the data¬ 
base tables referenced by the queries and the query structure 
itself. For our example, we shall first present, by illustration, 
the source database contents, the CONVERT operations 
used on the source database and the newly formed target 
database contents. The query will then be put through the 
analysis process and the results of the stage-by-stage anal¬ 
ysis will be given. The Stage 5 results are only summarized. 
They show the outcome of tracing the procedure steps and 
functions. 
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Source Database 


EMPLINFO 


EMP# 

DEPT 

PHONE 

EXPER 

AGE 

SEX 

RACE 

1 

TOYS 

25791 

3 

20 

M 

W 

2 

MENS 

28192 

3 

45 

F 

N 

3 

LADIES 

29999 

1 

36 

M 

A 

4 

SHOES 

28181 

1 

29 

M 

W 



MGRINFO 




EMP# 

DEPT 

PHONE 

EXPER 

AGE 

SEX 

RACE 

100 

MENS 

33456 

6 

40 

M 

A 

200 

TOYS 

37878 

6 

38 

M 

N 

300 

LADIES 

35512 

6 

62 

M 

W 

400 

SHOES 

35418 

7 

56 

M 

W 



EMPSIDAL 




EMP# 

NAME 

SALARY COMM 

DEPEND 

1 

TERRY KERRIGAN 

12000 


4 

1 

2 

NITA CALLAHAN 

13000 


3 

1 

3 

JEFF ICARDI 

08000 


1 

2 

4 

TIM MALONE 

07000 


1 

2 



MGRIDSAL 




EMP# 

NAME 

SALARY BENEFITS DEPEND 

100 

CHRIS CLEARY 

15000 


1 

0 

200 

JOHN HAUSER 

15000 


1 

0 

300 

SCOTT RAYBURN 

15000 


1 

1 

400 

PAUL GIOIA 

18000 


2 

8 


DEPTSALES 


DEPT 

SALES 

INVENTY 

MENS 

36009 

1024 

TOYS 

10112 

5890 

LADIES 

29001 

6950 

SHOES 

12152 

0500 


MONTHBILLS 


COMPANY 

BILL 

USAGE 

CODE 

KENNEYS 

04560 

00010 

A 

SCHAFFERS 

01130 

00195 

B 

LEVIS 

00050 

00080 

A 

KTEL 

10905 

01059 

C 



SERVICES 


ITEM 

COMPANY 

ADDRESS 

PHONE 

BILLDATE 

01 

KENNEYS 

GAINESVILLE 

3781125 

01 

02 

SCHAFFERS 

MIAMI 

4542121 

08 

03 

LEVIS 

ATLANTA 

3771156 

15 

04 

KTEL 

MIAMI 

5518907 

25 
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SPECITEM1 


ITEM 

DEPT 

CODE1 

PRICE 

QTY 

01 

MENS 

001 

00215 

30 

02 

TOYS 

001 

00389 

20 

03 

LADIES 

033 

00111 

40 

04 

SPORTS 

033 

04450 

15 

05 

AUTO 

010 

00889 

67 


SPECITEM2 


CODE2 ADVERTZ SUPPLIER 


AA 1 101 

BA 1 111 

BB 2 222 

CD 4 321 

Source Database Convert Operations 

(1) EMPIDENT EMPIDS AL(EMP# ,N AME) 

(2) MGRIDENT(MGR# ,NAME)<-MGRIDSAL(EMP# ,NAME) 

(3) EMPIDSAL(EMP#,SALARY,COMM) 

(4) MGRIDSAL(MGR#,SALARY,BENEFITS)«-MGRIDSAL(EMP#,SALARY,BENEFITS) 

(5) SELECT(ITEM,DEPT,CODE 1,PRICE,QTY FROM SPECITEM1:CODE1 = ‘001’ OR CODE1 = ‘033’) 

(6) GRAFT(MONTHBILLS ONTO SERVICES: MONTHBILLS.COMPANY= SERVICES.COMPANY) 

(7) SERVICEl(COMP ANY,AMTDUE,USAGE,CODE,ITEM,LOCATION,PHONE,BILLDATE)^SERVICES 

(COMPANY,BILL,USAGE,CODE,ITEM,ADDRESS,PHONE,BILLDATE) 

(8) CONCAT(SPECITEM2 ONTO SPECITEM1 AT QTY) 

(9) STAFFINFO<-MERGE(EMPLINFO,MGRINFO) 

(10) SORT(DEPTSALES BY INVENTY) 

(11) STAFFINFO<—CASE(STAFFINFO. SEX=‘M’ ,‘F’) 

(SELECT(EMP#,DEPT,PHONE,EXPER,AGE,‘MALE’,RACE FROM STAFFINFO), 
SELECT(EMP#,DEPT,PHONE,EXPER,AGE,‘FEMALE’,RACE FROM STAFFINFO)) 


Target Database 


EMPIDENT 


EMP# 

NAME 

1 

TERRY KERRIGAN 

2 

NITA CALLAHAN 

3 

JEFF ICARDI 

4 

TIM MALONE 


MGRIDENT 

MGR# 

NAME 

100 

CHRIS CLEARY 

200 

JOHN HAUSER 

300 

SCOTT RAYBURN 

400 

PAUL GIOIA 


EMPIDSAL 

EMP# 

SALARY COMM 

1 

12000 4 

2 

13000 3 

3 

08000 1 

4 

07000 1 
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MGRIDSAL 


MGR# 

SALARY 

BENEFITS 

100 

15000 

1 

200 

15000 

1 

300 

15000 

1 

400 

18000 

2 


SPECITEM1 


ITEM DEPT 

CODE1 

PRICE 

QTY 

CODE2 

ADVERTZ SUPPLIER 


01 MENS 

001 

00215 

30 

AA 

1 

101 


02 TOYS 

001 

00389 

20 

BA 

1 

111 


03 LADIES 

033 

00111 

40 

BB 

2 

222 


04 SPORTS 

033 

00450 

15 

CD 

4 

321 





SERVICE 1 




COMPANY AMTDUE 

USAGE 

CODE 

ITEM 

LOCATION 

PHONE 

BILLDATE 

KENNEYS 

04560 

0010 

A 

01 

MIAMI 

3781125 

01 

SCHAFFERS 

01130 

0195 

B 

02 

ATLANTA 

4542121 

08 

LEVIS 

00050 

0080 

A 

03 

GAINESVILLE 

3771156 

15 

KTEL 

10905 

1059 

C 

04 

MIAMI 

5518907 

25 


STAFFINFO 

EMP# 

DEPT 

PHONE 

EXPER 

AGE 

SEX 

RACE 

1 

TOYS 

25791 

3 

20 

MALE 

W 

2 

MENS 

28192 

3 

45 

FEMALE 

N 

3 

LADIES 

29999 

1 

36 

MALE 

A 

4 

SHOES 

28181 

1 

29 

MALE 

W 

100 

MENS 

33456 

6 

40 

MALE 

A 

200 

TOYS 

37878 

6 

38 

MALE 

N 

300 

LADIES 

35512 

6 

62 

MALE 

W 

400 

SHOES 

35418 

7 

56 

MALE 

W 


DEPTSALES 


DEPT 

SALES 

INVENTY 

SHOES 

12152 

0500 

MENS 

36009 

1024 

TOYS 

10112 

5890 

LADIES 

29001 

6950 


STAGE 1: Translation Dictionary Development 


TNR 


SOURCE TABLE NAME TARGET TABLE NAME 


EMPIDSAL 

MGRIDSAL 

MONTHBILLS 

SERVICES 

SPECITEM2 

EMPINFO 

MGRINFO 


EMPIDENT 

MGRIDENT 

SERVICE 1 

SERVICE 1 

SPECITEM1 

STAFFINFO 

STAFFINFO 
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DNR 

SOURCE DOMAIN 

TARGET 

NAME 

DOMAIN NAME 

EMP# 

MGR# 

BILL 

AMTDUE 

ADDRESS 

LOCATION 


TDU 



EMP 

IDENT 

MGR 

IDENT 

EMP 

IDSAL 

MGR 

IDSAL 

SPEC 

ITEM1 

SER 

VICE1 

STAFF 

INFO 

DEPT 

SALES 

EMP# 

1 

0 

1 

0 

0 

0 

1 

0 

NAME 

1 

1 

0 

0 

0 

0 

0 

0 

MGR# 

0 

1 

0 

1 

0 

0 

0 

0 

SALARY 

0 

0 

1 

1 

0 

0 

0 

0 

COMM 

0 

0 

1 

0 

0 

0 

0 

0 

BENEFITS 

0 

0 

0 

1 

0 

0 

0 

0 

ITEM 

0 

0 

0 

0 

1 

1 

0 

0 

DEPT 

0 

0 

a 

0 

1 

0 

1 

1 

PRICE 

0 

0 

0 

0 

1 

0 

0 

0 

QTY 

0 

0 

0 

0 

1 

0 

0 

0 

CODE2 

0 

0 

0 

0 

1 

0 

0 

0 

ADVERTZ 

0 

0 

0 

0 

1 

0 

0 

0 

SUPPLIER 

0 

0 

0 

0 

1 

0 

0 

0 

COMPANY 

0 

0 

0 

0 

0 

1 

0 

0 

AMTDUE 

0 

0 

0 

0 

0 

1 

0 

0 

USAGE 

0 

0 

0 

0 

0 

1 

0 

0 

CODE 

0 

0 

0 

0 

0 

1 

0 

0 

LOCATION 

0 

0 

0 

0 

0 

1 

0 

0 

PHONE 

0 

0 

0 

0 

0 

1 

1 

0 

BILLDATE 

0 

0 

0 

0 

0 

1 

0 

0 

EXPER 

0 

0 

0 

0 

0 

0 

1 

0 

AGE 

0 

0 

0 

0 

0 

0 

1 

0 

SEX 

0 

0 

0 

0 

0 

0 

1 

0 

RACE 

0 

0 

0 

0 

0 

0 

1 

0 

SALES 

0 

0 

0 

0 

0 

0 

0 

1 

INVENTY 

0 

0 

0 

0 

0 

0 

0 

1 

CODE1 

0 

0 

0 

0 

1 

0 

0 

0 


TCI 











CON 

CASE 










ELIM 

SOLI 

ASSIGN 


ASSIGN 

C/E 

SELECT 

SLICE 

GRAFT 

CONCAT 

MERGE 

SORT 

DUP 

DATE 

MENT 

EMPLINFO 

1 

0 

0 

0 

0 

0 

1 

0 

0 

0 

1 

MGRINFO 

1 

0 

0 

0 

0 

0 

1 

0 

0 

0 

1 

EMPIDSAL 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

MGRIDSAL 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

DEPTSALES 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

MONTHBILLS 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

SERVICES 

I 

1 

0 

0 

1 

0 

0 

0 

0 

0 

0 

SPECITEM1 

0 

0 

1 

0 

0 

1 

0 

0 

0 

0 

0 

SPECITEM2 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 
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CAD 


TARGET TABLE NAME 

TARGET 
DOMAIN NAME 

SOURCE 
DOMAIN VALUE 

TARGET 

DOMAIN VALUE 

STAFFINFO 

SEX 

M 

MALE 

STAFFINFO 

SEX 

F 

FEMALE 


GCI 

SOURCE TABLE NAME ONTO TABLE NAME 

MONTHBILLS SERVICES 

SERVICES SERVICES 

SOC 

SPECIAL CONDITION 

SOURCE TABLE NAME SOURCE DOMAIN NAME VALUE OPERATOR 

SPECITEM1 CODE1 001 

SPECITEM1 CODE1 033 

STAGE 2: Source Database Query Input* 

LI,SI_SELECT NAME 

FROM EMPIDSAL 

WHERE EMP# = 

L2,S1_SELECT EMP# 

FROM EMPLINFO 

WHERE EXPER > 1 

AND AGE < 36 

OR RACE = A’ 

AND SALARY > 9000 

* This query is recognized as one of the possible structures capable of being operated upon by the formal conversion 
algorithms. 

STAGE 3: Query Translation Preparation 

(1) The database table of the LEVEL 1, SEGMENT 1, basic mapping was acted upon by both the ASSIGNMENT and 
COMPONENT EXTRACTION CONVERT operations. Thus perform: 

PROCEDURE 1 

(2) The database table of the LEVEL 2, SEGMENT 1 basic mapping was acted upon by ASSIGNMENT, MERGE and 
CASE ASSIGNMENT. Thus perform the following algorithms in the order specified: 

PROCEDURE 1 
PROCEDURE 3 

STAGE 4: Query Pre-Translation Reference Table Development 
LEVEL 1, SEGMENT 1: 


REFTB1 REFTB2.0 


WHERE CLAUSE 
DOMAIN(S) 

TARGET DATABASE 
TABLE 

AND CLAUSE 
DOMAIN(S) 

TARGET DATABASE 
TABLE 

EMP# 

EMPIDSAL 

SALARY 

EMPIDSAL 

EMP# 

EMPIDENT 

SALARY 

MGRIDSAL 

EMP# 

DEPTSALES 



MGR# 

MGRIDENT 



MGR# 

MGRIDSAL 
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REFTB3 


WHERE CLAUSE 
DOMAIN(S) 

REFTB2.0 AND 
CLAUSE DOMAIN(S) 

TARGET DATABASE 
TABLE 

EMP# 

SALARY 

EMPIDSAL 

MGR# 

SALARY 

MGRIDSAL 

REFTB4.0 

REFTB6 


OR CLAUSE 

TARGET DATABASE 

SELECT CLAUSE 

TARGET DATABASE 

DOMAIN(S) 

TABLE 

DOMAIN 

TABLE 


empty 

NAME 

EMPIDENT 



NAME 

MGRIDENT 


LEVEL 2, SEGMENT 1: 


REFTB1 


REFTB2.0 


WHERE CLAUSE 

TARGET DATABASE 

AND CLAUSE 

TARGET DATABASE 

DOMAIN(S) 

TABLE 

DOMAIN(S) 

TABLE 

EXPER 

STAFFINFO 

AGE 

STAFFINFO 


REFTB3 

WHERE CLAUSE 

REFTB2.0 AND 

TARGET DATABASE 

DOMAIN(S) 

CLAUSE DOMAIN(S) 

TABLE 

EXPER 

AGE 

STAFFINFO 


REFTB4.0 


REFTB6 


OR CLAUSE 

TARGET DATABASE 

SELECT CLAUSE 

TARGET DATABASE 

DOM AIN(S) 

TABLE 

DOMAIN 

TABLE 

RACE 

STAFFINFO 

EMP# 

EMPIDENT 



EMP# 

EMPIDSAL 



EMP# 

STAFFINFO 



MGR# 

MGRIDENT 



MGR# 

MGRIDSAL 


REFTB5 


WHERE CLAUSE 
DOMAIN(S) 

REFTB2.0 AND 
CLAUSE DOMAIN 

REFTB4.0 OR 
CLAUSE DOMAIN 

TARGET DATABASE 
TABLE 

EXPER 

AGE 

RACE 

STAFFINFO 


STAGE 5: Formal Translation Algorithm 

Query Basic Mapping Processing Order: 

LI,SI 
L2,S1 

*BEGIN PROCESSING OF LI,SI BASIC MAPPING: 

SELECT NAME 
FROM EMPIDSAL 

WHERE EMP# = 

AND SALARY > 9000 
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PROCEDURE 1: 




TEST 

RESULT 

ACTION 

STEP 1: 

INRMP(L,S,WHERE) 

TRUE 

NONE 


REFTBL2(L,S,1) 

FALSE 

PROCEED 

STEP 2: 

CLAUSE(L,S AND) 

TRUE 

N=2.0 

J=1 

NONE 


INRMP(L,S,AND) 

FALSE 


REFTBL2(L ,S ,2.0) 

FALSE 

PROCEED 

STEP 3: 

CLAUSE(L,S,OR) 

FALSE 

NONE 

STEP 4: 

OUTMP(L,S) 

FALSE 

NONE 


DOMVAR(L,S,SELECT) 

FALSE 

NONE 


COMNTAB(L,S,6,19) 

TRUE 

SET REFTB20 = REFTB6 

STEP 5: 

REFTBL2(L,S,1) AND REFTBL3(L,S,2.0,2.9) 

FALSE 

NONE 


REFTBL2(L,S,3) 

FALSE 

PROCEED 

STEP 6: 

REFTBL2(L,S,5) 

TRUE 

NONE 


REFTBL3(L,S,4.0,4.9) 

TRUE 

SET REFTB5 = REFTB3 

STEP 7: 

COMNTAB2(L,S,5,20) 

FALSE 

NONE 


OUTMP(L,S) 

FALSE 

NONE 


REFTBL2(L,S,5) 

FALSE 

NONE 


REFTBL2(L,S,20) 

FALSE 

K=l, M= 1 RESTRCT3(L,S) 


TNR2(EMPIDSAL,EMPIDENT) 

TRUE 

PROCEED 

LI,SI PROCESSING RESULTS: 



SELECT NAME 

FROM EMPIDENT 

WHERE EMP# = 




SELECT EMP# 

FROM EMPIDSAL 

WHERE EMP# = 

AND SALARY > 9000 



*BEGIN PROCESSING OF L2,S1 BASIC MAPPING: 



SELECT EMP# 

FROM EMPLINFO 

WHERE EXPER > 1 

AND AGE < 36 



OR 

RACE = A’ 



PROCEDURE 1: 




TEST 

RESULT 

ACTION 

STEP 1: 

INRMP(L,S,WHERE) 

FALSE 

NONE 


REFTBL2(L,S,1) 

FALSE 

PROCEED 

STEP 2: 

CLAUSE(L,S,AND) 

TRUE 

N=2.0 

J=1 

NONE 


INRMP(L,S,AND) 

FALSE 


REFTBL2(L,S,2.0) 

FALSE 

PROCEED 

STEP 3: 

CLAUSE(L,S,OR) 

TRUE 

N=4.0 

J=1 

NONE 


INRMP(L,S,OR) 

FALSE 


REFTBL2(L,S,4.0) 

FALSE 

PROCEED 

STEP 4: 

OUTMP(L,S) 

TRUE 

NONE 


[SELDOM( 1) ,L,S]=[OMRCD,L,S] 

TRUE 

CLEAR ALL TUPLES OF REFTB6 WHERE 
THE DOMAIN NAME # EMP#’. SET 

REFTB20 = REFTB6. 

STEP 5: 

REFTBL2(L,S,1) AND REFTBL3(L,S,2.0,2.9) 

FALSE 

NONE 


REFTBL2(L,S,3) 

FALSE 

PROCEED 
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STEP 6: REFTBL2(L,S,5) FALSE PROCEED 

STEP 7: COMNTAB2(L,S,5,20) TRUE ENTTB2(L,S,Q.C.T.,5,20) 

RESTRCT1(L,S) 

RESTRCT2(L,S) 


PROCEDURE 3: 



TEST 

RESULT 

ACTION 

STEP 1: 

OUTMP(L,S) 

TRUE 

NONE 


CAD2(STAFFINFO,EMP#) 

FALSE 

NONE 


CAD2(EMPIDSAL,EMP#) 

FALSE 

PROCEED 

STEP 2: 

CLAUSE(L,S, WHERE) 

TRUE 

NONE 


NOT(INRMP(L,S,WHERE)) 

TRUE 

NONE 


CLSOPR(L,S, WHERE, =) OR CLSOPR(L,S, WHERE,#) 

FALSE 

MARK1=0 

N=1 

NONE 


CAD2(STAFFINFO,EXPER) 

FALSE 


MARXIAN and MARK1#0 

FALSE 

PROCEED 

STEP 3: 

CLAUSE(L,S,AND) 

TRUE 

1 = 1 


NOT(INRMP(L ,S, AND)) 

TRUE 

NONE 


CLSOPR(L,S,AND,=) OR CLSOPR(L,S,AND,#) 

FALSE 

MARK1=0 
N= 1 

NONE 


C AD2(ST AFFINFO, AGE) 

FALSE 


MARXIAN AND MARK1 # 0 

FALSE 

PROCEED 

STEP 4: 

CLAUSE(L,S,OR) 

TRUE 

1=1 


NOT(INRMP(L,S,OR)) 

TRUE 

NONE 


CLSOPR(L,S,OR,=) OR CLSOPR(L,S,OR,#) 

TRUE 

NONE 


DOMVAR(L,S,OR) 

FALSE 

NONE 


CAD2(STAFFINFO,RACE) 

FALSE 

PROCEED 

STEP 5: 

FUNCTN(L,S,SELECT) 

FALSE 

NONE 


ARITHOP(L,S,SELECT) 

FALSE 

PROCEED 

STEP 6: 

FUNCTYPE(L,S, 1, WHERE,SET) 

FALSE 

PROCEED 

STEP 7: 

CLAUSE(L,S,AND) 

TRUE 

1 = 1 


FUNCTYPE(L,S,1,AND,SET) 

FALSE 

PROCEED 

STEP 8: 

CLAUSE(L,S,OR) 

TRUE 

1=1 


FUNCTYPE(L,S, 1,OR,SET) 

FALSE 

PROCEED 


L2,S1 PROCESSING RESULTS: 

SELECT EMP# 

FROM STAFFINFO 
WHERE EXPER > 1 
AND AGE < 36 
OR RACE = ‘A’ 


STAGE 6: New Target Database Query 


SELECT 

NAME 



FROM 

EMPIDENT 



WHERE 

EMP# = 




SELECT 

EMP# 



FROM 

EMPIDSAL 



WHERE 

EMP# = 




SELECT 

EMP# 



FROM 

STAFFINFO 



WHERE 

EXPER >1 



AND 

AGE < 36 



OR 

RACE = ‘A’ 


AND 

SALARY > 

9000 
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CONCLUSION AND FUTURE RESEARCH 

Although the data independent property of a well-designed 
DBMS allows many types of database organization and re¬ 
structuring at both logical and physical levels without caus¬ 
ing changes to the application programs, there are situations 
where modification of application programs is unavoidable 
and is more beneficial in terms of system efficiency than 
using the mapping software associated with the schema and 
the subschema. As illustrated and argued in an earlier sec¬ 
tion, drastic reorganizations of the physical database, 
changes of the semantic associations among data entities 
and addition and deletion of semantic associations and en¬ 
tities often entail changes to the schema, the subschema and 
therefore the application programs of a DBMS. 

The existing data sublanguages, SEQUEL is an example, 
depend heavily upon the data model and the data submodel 
used. The sublanguage statements make direct references to 
the domain names, entity relation and association relation 
names and, to a certain degree though different from model 
to model, contain access path specifications to the data to 
be searched or processed. Thus, most changes made in a 
subschema would immediately terminate the life of appli¬ 
cation programs. To really increase the life span of appli¬ 
cation programs, it is necessary to make the data sublan¬ 
guage more independent of data models and submodels. 
How we can achieve this in language design remains to be 
investigated. 

At this stage of our investigation, we have worked out the 
algorithm for translating limited yet general language pat¬ 
terns in SEQUEL to account for the database changes in¬ 
troduced by CONVERT operators. This algorithm is de¬ 
scribed in four procedures written in PL/1 like statements. 
These procedures have been tested based on hand simula¬ 
tion of a large number of queries running against target 
databases produced by different sets of CONVERT state¬ 
ments. No actual coding of these procedures have been 
carried out because we believe that it will not add anything 
more to what we already can learn from the present work 
and it is not necessary for meeting our objectives stated in 
the Introduction at this stage of research. Based on our work 
and analysis in a somewhat restricted scope, we are quite 
confident to conclude that it is possible to implement a 
conversion system to either convert sublanguage statements 
automatically or provide assistance to the translation analyst 
in identifying the statements which need to be modified or 
the possible loss of data when running the old program 
against the new database. This can be done if the database 
translation task is carried out orderly by a well-defined da¬ 
tabase conversion language and the sublanguage used is well 
structured like SEQUEL. In our work, we found that the 
data tables in the translation dictionary are adequate for 
query conversions and the database changes due to all the 
CONVERT operators can be accounted for by query con¬ 
versions. The algorithm worked out in this work does not 
have any general application other than the conversion of a 
subset of SEQUEL queries. Nevertheless, it is believed that 
the general methodology and analysis process presented can 
be useful for future works on other languages even though 


it is clear that the idiosyncracies of the languages (the sub¬ 
languages as well as the data mapping languages) have to be 
accounted for. 

Besides the previously mentioned problem of designing a 
data language which is independent of the data models and 
submodels, the following problems seem to be worthwhile 
investigating. First, the access paths to the desired data in 
the source database can be very much different from the 
paths in the target database. In general, there are many 
access paths to the desired data. The selection of the short¬ 
est access path for forming the target query, i.e., the optim- 
ation of access path selection, needs to be further re¬ 
searched. Second, the selection of a proper access path that 
will be consistent with the semantics of the new database is 
an interesting problem. Any transformation of the source 
query not only has to preserve the original meaning of the 
query but also has to incorporate the data conditions based 
on which the target data files are formed. Some way of 
proving the correctness of the converted queries based on 
the semantics of the source queries and the target database 
would be a task of great importance. 

Another problem under our current investigation is based 
on the following observation. The basic semantic patterns, 
such as one-one, one-many, many-many associations be¬ 
tween/among entities, existing in different database systems 
are often the same even though different systems may use 
a different data model for user-database interface. In a given 
model, the representation of a basic semantic pattern is more 
or less fixed (e.g., a many-to-many association is repre¬ 
sented by two entity relations and an association relation in 
the relational model, 5 by two owner-coupled sets defined 
over two record types (for two entities) and a link record 
type in DBTG’s model 3,4 and by two physical databases with 
proper logical parent pointers to support two logical views 
of the n:n association in IMS 12 ). Since the sublanguage 
statements for processing the entities and their association 
depend upon the data representation as defined in the sub¬ 
schema, the access paths to the desired data in each basic 
semantic pattern should be restricted. The sublanguage pat¬ 
terns for carrying out the access paths can be exhaustively 
delineated. When a semantic pattern of data has been 
changed in a database translation, it should be possible to 
determine first the proper representation of the new data at 
the subschema level and then the sublanguage patterns 
which will do the same job as the source language pattern 
for the source data semantics. Furthermore, it should be 
possible to determine the pattern of data access in one 
source language and generate the corresponding pattern of 
data access in another language once the same data semantic 
pattern has been identified. Investigation of a generalized 
program conversion system and of the problem of converting 
application programs in one database system to that in an¬ 
other system is in progress. Research results along this line 
of thought are reported in References 27 and 28. 
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BACKGROUND 

Data processing installations preparing to undergo a system 
transition are faced with several difficulties; prominent 
among these being the transition of the current workload to 
a new system. 

A report by the General Accounting Office 1 published in 
September 1977, states that the annual Federal Government 
cost of modifying computer programs to enable them to 
execute correctly on a computer different from the one for 
which they were originally devised is estimated at more than 
$450 million. Comparable industry-wide figures are not 
available, but it is reasonable to assume that the overall cost 
of software conversion is significant. Furthermore, this is a 
non-productive cost; conversion per se results in no direct 
improvement in an organization’s ability to fulfill its mission. 

Research and development efforts are under way at sev¬ 
eral universities and research laboratories to determine ways 
and means of producing portable software, i.e., software 
which is machine and configuration independent over a set 
of computer installations (e.g., see References 2 and 3). At 
the same time, industry is reacting to the problem in a 
variety of ways, including a softening of architectual differ¬ 
ences (e.g., there are some half dozen IBM 370 “deriva¬ 
tives”) and an improvement in emulation capabilities. Until 
such efforts bear practical fruits, data processing organiza¬ 
tions will periodically be faced with the prospect of a soft¬ 
ware conversion effort. Such an effort is invariably faced 
with distaste and apprehension. 

These are several reasons why system conversion is such 
a disruptive process. First of all, programmers must be 
shifted from their regular assignments to the conversion 
task. This is true whether or not an outside contractor is 
used to assist in the conversion. Proper conversion requires 
documentation, and old documentation is often found to be 
inadequate, even in well managed installations. Aggravating 
this condition is the fact that the programmers who originally 
coded the system are frequently no longer with the organi¬ 
zation. It is also a sad fact that very few software develop¬ 
ment specifications include provisions for portability, even 
though techniques for reducing the impact of system changes 
on software are known and documented. Finally, conversion 
often will take place in conjunction with the implementation 
of a new system, thereby adding to the concomitant disrup¬ 
tion. 


Conversion is a many-faceted enterprise. An organization 
preparing itself for a conversion needs to consider not only 
the conversion process itself but also its management, the 
technical problems to be faced, the relationship of the con¬ 
version costs to the cost evaluation of the new systems 
being considered, the procurement of conversion support 
services, cost and time estimates, conversion alternatives 
(such as emulation), and training of personnel. A complete 
discussion of all of these topics is beyond the scope of any 
single paper. We will concentrate therefore on the descrip¬ 
tion of the conversion process itself and the technical diffi¬ 
culties associated with it. The emphasis will be on what 
needs to be done rather than on how to do it. This emphasis 
reflects a value judgment on the part of the author; namely, 
that most of the ignorance regarding conversion has to do 
with the process itself. All too often organizations mistak¬ 
enly liken conversion to development, fail to plan and pre¬ 
pare properly, and invariably are overly parsimonious in the 
allocation of resources to the conversion effort. 

Before proceeding, the following definitions are provided 
in order to set the scope of this paper: 

“ Conversion : By conversions we mean any change made 
to a program or system of programs solely for the purpose 
of enabling such a program or system to execute correctly 
on a computer different from the one for which they were 
originally devised. Translation refers to a largely auto¬ 
mated process of conversion in which the original pro¬ 
grams themselves serve as adequate specifications for the 
new programs to be produced. Recoding is similar to 
translation except that the process is largely manual. Re¬ 
programming refers to a conversion which may entail a 
system redesign (e.g., batch to on-line) but no significant 
functional redesign. Redesign refers to a conversion effort 
which involves functional redesign and is therefore akin 
to new development.” 

CONVERSION PROJECT OVERVIEW 

Preparation 

The first step in the preparation for conversion is the 
requirements analysis. A review of the planned differences 
between existing systems and the converted system is par- 
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ticularly important if the language dialect being converted 
to has new language modules (which may be fruitfully used 
in the converted system), or major changes to input-output 
modules. It will also be necessary to identify the degree to 
which the compiler being used differs, in its implementation 
of the language, from the standard specifications for that 
language. 

Tasks, schedules, resource requirements, and end prod¬ 
ucts must be identified. Schedules and resource require¬ 
ments are particularly difficult to gauge. It is generally a 
good idea to obtain contractor support in preparing time and 
resources estimates, since broad experience in conversion 
is required in making such estimates. The review of existing 
programs may reveal some which will require no conversion. 
It is doubtful that there will be many of these. The rest must 
be collected together with accompanying files and docu¬ 
mentation, and placed in the hands of a conversion team. 

Finally, the specifications of system changes must be de¬ 
fined. Data file changes may be required; file and record 
sizes, field contents, file organization, access keys, sort 
keys, access methods, storage media, and labeling conven¬ 
tions are all likely candidates for change. The conversion 
will present an opportunity for some needed system restruc¬ 
turing; programs to be combined, intermediate files elimi¬ 
nated, or sort/merges which may be deleted if the restruc¬ 
turing involves a shift from tape to DASD residence for 
certain files. Also carefully specified must be any processing 
logic changes which are necessitated by differing language 
dialects. 

Software tools must be identified and developed. Software 
must be available to load data, copy programs, create ex¬ 
tracted versions of test data, perform data and file conver¬ 
sions, compare tests results for validity, and measure tests 
for reliability. Procedures must be developed and controls 
and quality assurance standards must be specified. 

Programs must be collected in a uniform way to ensure 
that the correct version (release) on every program is being 
converted. The software indicated above may be used to 
create test libraries and to control this step. A procedure for 
maintenance change inclusion must be developed, and a 
reference base for changes is hereby established. 

Adequate test data must be prepared which will exercise 
an acceptable portion of the converted programs. Each pro¬ 
gram should have a set of known inputs which produces 
known outputs in order to validate each program. Ideally, 
unit test data can also be used for system testing. 

Finally, all related materials must be collected. This in¬ 
cludes program and system documentation (flowcharts, nar¬ 
ratives, run-books, data layouts), inventories of files and 
programs, source listings, program assemblies (listings), and 
a directory of every item’s physical location. 

Production 

As subsystems become ready for actual conversion the 
translation process begins. This is true even if systems are 
to be eventually modified. That is, the conventional wisdom 
is that program translation (i.e., a one-for-one, or close to 


it, conversion) should precede any modification. This is 
done to avoid intermingling, and thereby compounding any 
translation errors or effects with modification errors or ef¬ 
fects. The success of the conversion will be closely related 
to the adequacy of the controls which are applied during the 
production stage. Controls must be established for the re¬ 
ceipt, handling, and distribution of all materials, for the 
copying and analysis (to ensure the correctness of the copy 
process) of program tapes; and for the definition and use of 
job control language programs. 

The translation process itself will be partly automated. 
Many features of programming languages lend themselves 
to automatic translation, using commercially available or in- 
house developed utilities. Unfortunately, a large portion of 
the input-output coding will have to be hand-translated, and 
in some cases the process will necessarily have to be closer 
to modification than translation. 

Thorough unit and system tests should follow the trans¬ 
lation phase (and will have to be repeated after the modifi¬ 
cation phase, if any). It is generally advisable to desk-trace 
the programs in a gross way, i.e., through job control, 
housekeeping, and initial input statements. A monitor which 
intercepts and analyzes abnormal terminations would be an 
extremely useful tool during testing. The monitor should be 
capable of displaying the instruction causing the abnormal 
termination, the data being processed at that time, and of 
providing snapshots of selected data/program areas. Also 
useful would be a file-compare utility to determine the va¬ 
lidity of the outputs produced by the translated program and 
a monitor which could recognize units of untested code. 
Once a translated system has been successfully tested any 
required modifications can take place. These may include 
system restructuring (combination of common subroutines, 
sort/merge utilities, etc.), changes in logic, and changes in 
data files. 

Finally, the entire process must be thoroughly and care¬ 
fully documented. The precise form of the documentation 
will depend on the installation’s standards, but should in¬ 
clude at least the following: 

Converted source programs 

Flowcharts of the converted systems 

Listing of all job control language programs used 

Standard file labels 

File conversion parameters 

Operating instructions and technical notes 

Unit and system test reports 

The following is a step-by-step summary of how a pro¬ 
duction team should perform the translation of programs: 

a. Materials are received by the production team and 
processed by a Control Section. Each tape is analyzed 
to ensure readability, and copied to backup tapes. Test 
data are converted to target machine format. Standard 
job control code is generated. Task estimates and a 
schedule for the program translation are created by a 
resource management system. 

b. The source program is converted to the target language 
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by a multi-step process, using appropriate software 
tools. It has been found useful to first convert the 
source code to some intermediate language to permit 
standardized analysis and manipulation. This is partic¬ 
ularly true if the conversion is to involve several source 
and target languages. The eventual restructured inter¬ 
mediate language program is then converted into prop¬ 
erly formatted target language code. The target pro¬ 
gram listings and other documentation are collected 
and given to the project manager, who assigns the 
program to an analyst for completion of the documen¬ 
tation. 

c. Corrections are made to the target program, again 
using appropriate software tools. The program is com¬ 
piled until all diagnostics are resolved. Two program¬ 
mers should then desk check every line to verify logical 
equivalence to the source program. 

d. Testing begins with the aid of a cross-reference pro¬ 
gram, a tool to trap processing exceptions and allow 
continued processing, and a file compare program to 
verify output data equivalence. Unexecuted code is 
identified and carefully desk checked. 

e. Unreferenced code is located and identified. Old data 
and procedure names are replaced where required by 
new names. The source code is formatted to installa¬ 
tion standards to ensure the uniform appearance of all 
programs. 

f. When program translation is completed, system en¬ 
hancements are applied, and maintenance changes are 
identified and implemented. 

g. The completed programs are returned to the project 
manager for a quality control check. The material is 
then processed by a control section and prepared for 
shipment to the implementation group. Backup copies 
of programs are stored on tape, along with microfilmed 
copies of listings for future reference. 

Implementation 

Implementation of the converted system should not take 
place until the systems software to be used has reached a 
satisfactory level of stability (this will be a subjective deci¬ 
sion). The conversion manager should anticipate changes 
(upward) in the resources required to compile and execute 
the converted programs, and should allow for these changes 
in order to avoid serious degradation in throughput. 

Unit and integration testing should be repeated on con¬ 
verted programs and test data once these are installed on 
the target system. Once this testing is satisfactorily com¬ 
pleted, maintenance changes are applied and tested, prob¬ 
lems are corrected, and retesting is performed. This process 
is repeated until all tests are successfully passed. It should 
be noted that this inclusion of maintenance changes entails 
the generation and conversion of production test data to be 
used in the testing process. Note also that no further changes 
must be applied to the production programs at this time. 

In the meantime, the production database itself is con¬ 
verted and tested, and the operating system control language 


production stream is generated. Finally, production testing 
using the final version of programs, data, and control lan¬ 
guage programs takes place (acceptance testing), followed 
by an appropriate period of parallel testing. 

Software tools 

The description of the conversion phases included several 
references to software tools. A complete inventory of such 
tools is expensive, and this would be one of the factors 
which must be taken into consideration when contemplating 
an in-house conversion, since this inventory would not be 
of great value after the conversion. 

There are several ways in which a software support in¬ 
ventory could be characterized, the simplest of which is 
according to the three principal conversion stages in which 
they are used: 

a. Preparation: A file contents analyzer, a data extractor 
and modifier, and a data generator can all be used for 
creating test data. Utilities will also be required for 
creating backups of all programs, and for maintaining 
a current version of the software inventory to be con¬ 
verted and producing statistics (e.g., average program 
sizes) as required. 

b. Production: The production stage will require software 
to perform source code to intermediate code transla¬ 
tion, to analyze and restructure the intermediate code, 
to perform intermediate code to target code translation, 
and to translate test data files. Utilities will be needed 
to generate the operating system control stream and to 
apply code corrections to translated code. Addition¬ 
ally, software to produce cross-reference listings, to 
trap and identify exceptions, to identify unexecuted 
code, and to perform file comparisons will be needed 
for testing. A de-compiler or depatcher will be needed 
if the conversion is from an assembler language to a 
higher level language. 

c. Implementation: Software to validate the results of 
parallel testing, to identify and implement maintenance 
changes, and to convert the production data base is 
required. 

Additionally, software aids will be useful in the manage¬ 
ment of the conversion project. As a minimum, software 
tools are needed for resource management (identify pro¬ 
grams and categorize by source and content, estimate re¬ 
source requirements, and monitor progress), and standards 
enforcement (format programs to installation standards, re¬ 
place old names with standard names, etc.). 

Managing the conversion project 

A conversion project is only slightly different from any 
other software production project with respect to its man¬ 
agement. Careful planning is required, the project must be 
initiated and, once initiated, it must be controlled. Finally, 
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there is a completion phase. If there is a significant differ¬ 
ence between a conversion project management and a pro¬ 
duction project management it is one of emphasis; a con¬ 
version project requires (and allows for) more discipline and 
stricter adherence to procedures. If properly executed, a 
conversion is very much an assembly-line type of operation, 
where the total effort is broken down into well-defined tasks 
which are more dependent on experience and strict adher¬ 
ence to procedures than an innovation and ingenuity for 
their successful completion. This is true partly because of 
the high degree to which the conversion process can be 
automated. It should also be noted that many of the ground- 
rules for software production do not apply to conversion. 
For example, manpower and time are not generally inter¬ 
changeable in a software production project, but, within 
bounds of good taste, they are in a conversion project. 

Productivity rates also differ widely between conversion 
and development. Software development seems to proceed 
at some 12-20 lines per man-day for general application 
software, while conversion may proceed at rates as high as 
400 lines of code per man-day. 

A word of warning regarding project organization—a com¬ 
mon mistake is to make the conversion staff a part-time 
group which participates in conversion activities but whose 
members continue to report to their parent organizations. 
This is an ingenious way to make a mess of the conversion, 
particularly since it will be nearly impossible to attribute 
responsibility for the mess to any one person. 

The specific makeup and size of the conversion staff will 
vary with the conversion type and magnitude. The following 
membership, without specific quantities, is suggested for a 
large-scale project which will require contractor support. 
Following each staff category is an indication of the role of 
that category: 

a. Project Leader 

(1) Planning 

(2) Project initiation 

(3) Project control 

(4) Project termination 

b. Contracting—staff support advising on 

(1) Type of contract 

(2) Necessary clearances 

(3) Terms and conditions 

(4) Scheduling 

c. Operations 

(1) Present status and future needs 

(2) Performance specifications 

(3) Scheduling 

(4) Inventory 

(5) Computer resources 


d. Systems Programming 

(1) Inventory 

(2) Sizing a job 

(3) Performance specifications 

e. Application systems developers 

(1) Inventory 

(2) Sizing 

(3) Performance specifications 

f. Support programming—software tools for 

(1) Inventory 

(2) Quality control 

(3) Production 

(4) Testing 

g. Material Control 

(1) Quality assurance 

(2) Backup inventory 

(3) Materials transmittal 

h. Clerical—supports entire team 

i. Analysis & Programming 

(1) Production 

(2) Testing 

(3) Implementation 

Figures 1 and 2 suggest the organizational relationships 
among the staff components, and Appendix A lists the major 
tasks to be performed in the form of a checklist. 

CONVERSION PROBLEMS 
General 

A conversion project results in a very large volume of 
material, and this results in control programs. Staffing re¬ 
quired for the conversion project will not be needed at the 
culmination of conversion and will be diverted from on¬ 
going development and maintenance. Furthermore, there 
will be periods of peak requirements. This will create serious 
management problems. 

Machine time will also encounter periods of peak loads. 
This, unfortunately, conflicts with growing production due 
to the cutover of subsystems. Machine time availability 
takes on an inverse relationship to conversion requirements. 
In many cases, conversion requirements plus production 
requirements become more than the total of machine avail¬ 
ability. This causes costly delays in the conversion schedule. 

Low resources requirements estimates are caused by a 
lack of understanding of the conversion process. An esti- 
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o Systems Programmers o Systems Programmers 

o User 
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o Management 

Figure 1—Conversion project organization (Feasibility analysis and 
planning) 


mator may take a few programs, convert them, and attempt 
to extrapolate the requirements for the whole job. This pro¬ 
jection is inaccurate because it represents a straight line 
relationship (which never really exists) between conversion 
volume and resource requirements. This mistake is of course 
the same one which has plagued production time estimators 
for years. As in software production, size has a special effect 
on conversion time and costs which render a linear relation¬ 
ship invalid. For example, a competent programmer, prop¬ 
erly supported by a set of software tools, can convert 500 
lines of COBOL source code per day. This suggests that a 
10,000 line program could be converted in 20 days, whereas 
experience shows that some 36 days would in fact be re¬ 
quired, and that this number could go up to 104 days if the 
conversion is from an assembler language with complex file 
structures and large file volumes. Another program in esti¬ 
mating is the tendency to exaggerate the extremes by making 
simple tasks appear too simple, and difficult one too diffi¬ 
cult. 

Technical 

A complete discussion of all the technical problems of 
conversion is beyond the scope of this paper. What follows 
is a brief discussion of some of the differences in machine 
architectures and file structures which can impact conver¬ 
sion. 


Computer words vary in the number of characters that 
they contain causing problems in numerical accuracy and 
data movement. The accuracy program can be dealt with by 
increasing arithmetic precision, but this will increase storage 
requirements as well. The program in data movement is that 
programs can often refer to the storage of words or char¬ 
acters. When we translate these programs, the net effect is 
inconsistent treatment of data. A UNIVAC 1108 with a 36- 
bit word has six 6-bit characters. If we were converting to 
an IBM S/370, the 1108 program would move six characters 
in moving a word while the S/370 programs would move 
four characters in moving a word. To solve the program we 
have to determine whether or not the code refers to char¬ 
acter data. This process often involves lengthy analysis, 
which requires personnel time, reduces automation, and in¬ 
creases time and costs of conversion. 

Machine condition indicators are internal switches that 
are set for special conditions, e.g., overflow, zero divide, or 
invalid data. Problems arise when the source machine sets 
an indicator and the target machine either does not, or does 
so under slightly different conditions. For example, the 1401 
would treat a space as a zero when doing any arithmetic. 
An IBM 370 will cause an interrupt called a data exception 
(OC7) when any non-numeric data appear as an operand in 
an arithmetic statement. This condition must be corrected 
in the translated program. 

The format and the amount of information that is specified 
to define a file varies among languages. Some languages may 
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Figure 2—Conversion project organization (Post-planning) 


require very little information to be specified, while other 
languages will require a great deal of information like record 
length, block size, or labels. Often there is even a wide 
variance between programs written in the same language 
because a language will allow, but not require certain spec¬ 
ifications, or there will be different ways to define and spec¬ 
ify file information in the same language, e.g., standard 
macros, user macros, or no macros (user-coded I/O). 

The specifications of a file may be defined implicitly or 
explicitly. Explicit definitions are required in a COBOL 
program. An implicit definition might occur in an assembler 
program where a file is defined only by its use. Thus, the 
program and all references to the file must be analyzed to 
determine the appropriate file definitions. A similar problem 
also arises in data definition. A data item could be defined 
explicitly as a character field or not defined at all. All as¬ 
semblers allow a programmer to specify arithmetic and/or 
binary operations on any data field. The actual use of a data 
item defines its implicit attributes. These may differ from 
their defined attributes in both data type and field length. 
When there is a discrepancy, an alternate definition must be 
generated to accommodate the circumstances. This condi¬ 
tion is further complicated when implicit usages change be¬ 
cause of word size variances. 

No “standard” format exists for the recording of variable 
length records on tape or disk. Some of the ways in which 


the records are delineated are by delimiting records by spe¬ 
cial characters to indicate the stopping or starting point of 
each record, recording the variable length records as fixed 
length records by padding their length to a maximum con¬ 
sistent length, or inserting byte or word counts at the begin¬ 
ning of each record. 

File organization becomes a consideration on any nonse¬ 
quential file. Generally, indexed files contain pointers to 
indicate sequencing. Problems arise because variances are 
encountered in the location of the pointer, the format of the 
pointer, and the meaning of the pointer (some access meth¬ 
ods will use actual track addresses, while others will use a 
relative record number within the file). The conversion of 
an indexed file includes getting the disk file onto tape, con¬ 
verting the pointer to the new format, and reloading the file 
to disk. The manner in which an indexed file is processed 
usually varies by language and machine. The differences 
may involve module communication or linkage to the access 
method for the system. Many direct access applications are 
developed using an algorithm to compute and determine the 
location of records on disk. For example, the disk addressing 
techniques of a source machine may be determined by an 
algorithm based on bytes per record, records per track, or 
tracks per cylinder. When converting to a relative addressing 
technique, or just a disk file with different characteristics, 
one or all of the above disk parameters could change. 
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The functions of OPEN and CLOSE in a program will 
vary. A tape-oriented program may OPEN and CLOSE each 
tape reel as a separate file; this coding must be eliminated 
when converting to a concatenated or disk file. Almost all 
I/O systems have different techniques for handling I/O er¬ 
rors. They also have different error recovery routines. Dif- 
fermces in these routines must be examined and handled. 
Communication with error routines must also be dealt with. 

Data is frequently represented in different ways on the 
source and target machines. Often the differences can be 
easily dealt with, but representation differences can present 
significant problems. This for example, the BCD (Binary 
Coded Decimal) and EBCDIC (Extended Binary Coded 
Decimal) character sets have bit patterns for numbers (0-9) 
that have a greater binary value than the letters. Several 
other character sets do not maintain this same relationship 
and the bit pattern for the numbers has a lower value than 
for the letters. Conventions and requirements for represent¬ 
ing (or not representing) signs on numeric fields often differ 
between machines produced by the same vendor and always 
differ between machines produced by different manufactur¬ 
ers. For example, on some machines a negative amount on 
character numeric fields is indicated by a “a negative zone” 
and all other numbers are represented by “no zone”. On an 
IBM S/370 there are “-zones,” “+zones,” and “no zone” 
fields. On some systems a packed decimal number must 
always have a sign in the right four bits of the field. The 
Honeywell 8200 does not require a sign. If a sign is present, 
it can be from one to four bits long and will appear in the 
left of the word, not necessarily adjacent to the field. Var¬ 
ious Job Control Languages (JCL) are used to relate an 
application program to the hardware’s operating system. 
Each is usually unique to a computer system. All functions 
that a program performs with the source operating system 
must be converted to functions to be performed by the target 
program and/or operating system. Differences among JCL 
may exist with regard to inter-module communication. In¬ 
formation (data) is often passed between subroutines within 
a running program. Sometimes, the data is left in a “com¬ 
mon” area of memory by both the calling and the called 
subroutines. Communicated data must then be isolated and 
the address of the data passed between modules. Differences 
may also exist in the roles of the application program and 
the operating system. In the past, with programs running 
sequentially in computers, the operating system allowed the 
program to determine its own hardware usage, file alloca¬ 
tion, core allocation, and time requirements. When multi¬ 
programming, these functions are removed from the appli¬ 
cation programs and supplied by the operating system 
In addition to system differences, the differences created 
by individual organizational programming practices can cre¬ 
ate significant problems in a conversion effort. In fact, these 
represent the major factor impacting cost. Sometimes in¬ 
stallations using low level languages do not have stringent 
programming standards. This often results in non-uniform, 
inconsistent programs. COBOL or PL/I forces a certain 
degree of standardization. The transformation process must 
be designed to handle each of the different circumstances 
and to produce code in a standard high level language. In¬ 


sufficient, out-of-date or non-existent documentaton is a 
significant problem for most conversions. The conversion 
process does not require extensive documentation since we 
use the source program as the specification for the target 
program, but system level documentation (or job/file flow) 
is needed to restructure a system and record layouts are 
required for file conversions. Systems are often designed 
with specific hardware/software limitations or user options 
in mind. Examples are the use of checkpoint/restart for a 
program that runs too long, using overlays because of core 
limitations, using a disk file instead of a tape file because of 
a shortage of tape drives, or running a sort separately instead 
of as an exit in our application program. Many of these 
conditions can be changed to improve efficiency when the 
program or system is converted. 

Appendices B and C outline some specific sources of 
conversion difficulties which may occur in COBOL and 
FORTRAN programs, respectively. Reference 4 gives an 
excellent survey of transferability problems of programs 
written in these and other languages, and Reference 5 sug¬ 
gests techniques to be used in minimizing FORTRAN con¬ 
version problems. 

CONTRACTING FOR CONVERSION SERVICES 

Success in conversion is in large measure dependent on 
experience and few if any data processing organizations 
possess this experience. There is little reason why they 
should—conversion is not an ongoing enterprise. The ex¬ 
perience required is not of the “we have three professionals 
who have gone through a couple of conversions” type. 
Rather, experience, to be useful, must involve an entire 
team who has performed enough conversions to become 
proficient in the techniques and tools to be used, and pre¬ 
dictable in its productivity and performance quality. Such 
experience is most apt to be found in software services or 
conversion services contractors. Generally, it would be wise 
for organizations to avail themselves of contractor support 
for conversion. This is true particularly since, contrary to 
popular opinion, the software producers are not the best 
qualified people to convert the same software. If anything, 
they are the worst qualified, since they probably cannot 
resist the temptation to “improve” the system while con¬ 
verting it. 

Contracting methods vary greatly. The best, most sound, 
contracting is probably done by the U.S. Government under 
its Federal Procurement Regulations. 6 Useful guidelines can 
also be obtained from Reference 7. The following are offered 
as general factors to be considered. 

The first step in a procurement is the same as the first 
step in the conversion planning, namely, a requirements 
analysis. Once this is done, the contracting personnel can 
begin to plan for the type of contract which may be required 
by the specifications, the time required to prepare a request 
for proposals (RFP) to potential contractors, the time re¬ 
quired for negotiations and evaluation of proposals, and the 
time required to make an award. The availability of funds 
must also be determined at this time. Our experiences in- 
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dicate that the procurement may take from 22 to 38 weeks 
to complete. 

The technical staff can ease the burden of the contracting 
staff by preparing a complete procurement request (PR). 
This will include a thorough statement of the scope of work 
(supplies and services, etc.), reporting requirements, prop¬ 
erty or facilities to be provided to the contractor, and a list 
of prospective sources of support. Upon receipt of the PR 
the contracting officer can assemble his procurement staff 
which, for large or complex procurements may include ne¬ 
gotiators, a cost/price analyst, an inspector, legal counsel, 
and auditors. 

A complete discussion of contractor evaluation methods 
is beyond the scope of this paper, but the principal ones are 
listed below: 

• Cost only—pick low bidder among qualified ones. If the 
specifications are adequate this is not as bad a method 
as it is reputed to be. 

• Cost plus “desirables”—the basic bid price is modified 
up or down according to the quality or number of “non¬ 
mandatory” items a contractor purposes. This is a su¬ 
perficial way of giving “bonus points” which has little 
to recommend it in this author’s opinion. 

• Cost evaluation plus technical evaluation—it is common 
to attempt to combine “cost points” with “technical 
points” to determine a winner. This is usually moti¬ 
vated by a desire to substitute subjective judgment for 
a sound evaluation. One legitimate way of incorporating 
a technical evaluation into the overall evaluation 
scheme is to use the results of a technical review to 
determine a “technical competitive range,” to eliminate 
all bids falling outside this range, and to make the award 
to the lowest of the remaining bids. 

ESTIMATING CONVERSION COSTS 

Conversion costs for an in-house conversion (i.e., per¬ 
formed entirely by a staff of programmers from the organi¬ 
zation undergoing the conversion) cannot be accurately es¬ 
timated. The difficulties of estimating production efforts are 
well-known. Yet, software production is something which 
programmers do all the time. They do not do conversion all 
the time. Furthermore, the in-house staff does not have the 
tools and procedures required, and even if it acquires them 
it has little or no previous experience in using them. 

One can determine reasonable estimates of conversion 
costs if a contractor is used for the production stage, which 
is the most costly of the various conversion stages. The 
various tasks associated with planning and implementation 
can be itemized, resources required for each of these are 
estimated, and this results in a cost estimate for these por¬ 
tions of the conversion. 

Estimates for the production stage can be derived by 
reviewing past conversion efforts and prices bid on these 
efforts. The Federal COBOL Compiler Testing Service has 
compiled a substantial data base of cost and productivity 


figures. A complete description of the resulting cost model 
would require an entire paper itself, and will therefore not 
be presented here. A few figures can however given general 
indication of potential costs. 

• The cost of converting a line of COBOL code (to 
COBOL) may range from $.40 to $6.00, with $.40-2.00 
being a most likely band, and $.65-1.00 being a good 
average figure for reasonably clean COBOL programs 
requiring no extensive file restructuring on additional 
documentation. FORTRAN to FORTRAN conversion 
costs are similar to COBOL-COBOL costs. 

• Data on PL/I to PL/I conversion is scarce, but indica¬ 
tions are that such a conversion would be 10-20 percent 
costlier than a COBOL-COBOL conversion. Assembler 
code—COBOL/FORTRAN will cost from $2+ to $8 
per source line. 

• Productivity figures range from 300-500 lines per man- 
day in FORTRAN-FORTRAN or COBOL-COBOL 
down to 20-100 lines per man-day in an assembler- 
COBOL/FORTRAN conversion. 

• Extensive documentation (user guides, narratives, etc.) 
can cost up to 40 percent of the total per line cost. In- 
house costs for planning, preparation, and implemen¬ 
tation can be up to 50 percent of the total costs. 

There are also several qualitative observations which can be 
made from an analysis of a sizable conversion data base: 

• Conversion costs are more dependent on the source 
machine than on the target machine. 

• Knowledge of the application is not critical in perform¬ 
ing the conversion; but information regarding the ap¬ 
plication must be available as required. (Redesign is of 
course excepted here.) 

• Known complexity will not unduly influence costs, but 
complexity which comes as a surprisse can cause 
havoc. 

• None of the above holds for real-time systems. 

• Conversion of data base systems is not well under¬ 
stood. 

• Conversion estimates should not be attempted by peo¬ 
ple who do not have access to a sizable data base of 
information and who do not do this on a regular basis. 

• Cost models consisting of a handful of formulas based 
on a handful of parameters are worse than useless; they 
are dangerous because they give the impression of au¬ 
thority. 


CONCLUSION 

Conversion is a disruptive, largely non-productive process 
which must nevertheless be faced at some point in the life 
of a data processing organization. This paper has presented 
some ideas which might be of assistance to someone con¬ 
templating a conversion, the most important of which are 
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that conversion requires careful planning, and that it should 
not be attempted without the services of professionals. 
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APPENDIX A—SYSTEM CONVERSION CHECKLIST 
PLANNING 

Collect inventory and define scope of work 
Analyze differences between source and target systems 
Develop conversion plan and schedule 
Estimate resource requirements 
Assign conversion responsibilities 
PREPARATION 

Develop or acquire software tools required 
Collect and package programs 
Prepare adequate test data 
Create test output using test data 
Collect related materials 
PROJECT CONTROL 
Standardize conversion procedures 
Establish reporting requirements 
Monitor project status 
Assure quality of converted programs 
PRODUCTION 
Convert source code 
Code corrections required 
Prepare test job control 
Format programs to standard 
Convert test files 
TESTING 
Conduct unit test 
Conduct system test 
Conduct parallel test 
Ensure accuracy of converted programs 
Ensure test data adequacy 
INSTALLATION 
Implement maintenance changes 
Prepare production job control 


Convert production files 
Cutover to production 

APPENDIX B—COBOL PROGRAMMING PROBLEM 
SOURCES 
GENERAL: 

Requires operator intervention in processing 
Requires operator consol input(s) 

Has checkpoint/restart capabilities 
Contains an interface to database system 
Requires object code patches not incorporated into the 
current source 

CALLS to an assembler or other non-COBOL language 
subprogram 

Uses overlays or segmentation 
IDENTIFICATION DIVISION ENTRIES: 

Entries out of order with respect to the ANS COBOL 
Standard 

ENVIRONMENT DIVISION Entries 

FILE CONTROL 

I-O-CONTROL 

DATA DIVISION Entries 

FILE SECTION 

RECORDING MODE 

BLOCK CONTAINS 0 

USAGE IS COMP-1 short precision floating-point data 
COMP-2 long precision floating-point data 
COMP-3 internal decimal data (packed data) 

COMP-4 binary data 
WORKING-STORAGE SECTION 
Logic of the program expects certain initial values when 
data has not been initialized 
REDEFINES 

OCCURS DEPENDING ON 

Bit level data fields (non character aligned) 

Logical switches 
Logical masks 
Floating point literals 
Floating point fields 
Signed zero 

Unsigned numeric fields used in computations 

INDEX 

Subscripts 

Sort description SD-names 
LINKAGE SECTION 
Linkage entries 

COMMUNICATION SECTION 
Communication description CD-names 
REPORT SECTION 
Report writer description RD-names 
PROCEDURE DIVISION ENTRIES 
Program logic is sensitive to numeric precision 
Program logic dependent on the collating sequence, i.e., 
blank 

Logic dependent on HIGH VALUE or LOW VALUE 
Logic dependent on rounding or truncation of numeric 
results 
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Logical shifts or bit manipulating 
ALTER 
CLOSE 
COMPUTE 
EXAMINE 
GO TO DEPENDING 
Certain MOVE statements 


OPEN 


PERFORM (but not PERFORM . . . THRU . . .) 

SEARCH 

COPY 

TRANSFORM 

WRITE 

SORT 

LABEL 




An assessment of the technology for 
data- and program-related conversion* 


by JAMES P. FRY 

University of Michigan 
Ann Arbor, Michigan 

EDWARD BIRSS 

University of California 
Livermore, California 

PETER DRESSEN 

Honeywell Information Systems 
Phoenix, Arizona 

NANCY GOGUEN 

Bell Laboratory 
Piscataway, New Jersey 

MICHAEL KAPLAN 

Bell Laboratory 
Piscataway, New Jersey 


EUGENE LOWENTHAL 

MRI Systems Corporation 
Austin, Texas 

VINCENT LUM 

IBM Research 
San Jose, California 

ROBERT MARION 

Defense Communication Agency 
Reston, Virginia 

SHAMKANT NAVATHE 

New York University 
New York, New York 

STEVEN SCHINDLER 

University of Michigan 
Ann Arbor Michigan 


INTRODUCTION 

The scope of the conversion problem 

There are two factors that make conversion necessary: 
changes in users’ functional requirements and changes in 
their performance requirements. These changes may neces¬ 
sitate the acquisition of new software and hardware, which 
in turn, may require changes to the existing data programs. 
For example, the acquisition of a database management 
system to replace a file management system requires the 
integration of the original files into a database system and 
modification of the programs to interact with it. The replace¬ 
ment of a current DBMS with a new database system to 
provide additional functionality may require a different way 
of logically structuring the information (its data model) and 
a different kind of language interface. The establishment of 
a communication network between differing systems to im¬ 
plement data sharing will require dynamic (i.e., in real time) 
conversion of data between different nodes in the sense that 
the same item will repeatedly undergo conversion as it is 
needed, or alternatively, it requires conversion of programs 


* This is a report of the Conversion Technology Panel which met under the 
aegis of the ACM/NBS Workshop, “Database Directions—The Conversion 
Problem”. A complete report covering the other panels—Management Ob¬ 
jectives, User Experience, and Standarization—will be forthcoming from the 
National Bureau of Standards. 
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when they travel to different nodes to access data there. 
Even when the acquisition of new software or hardware is 
not warranted, changes in a users’ functional and perform¬ 
ance requirements can require data and program conversion. 

What makes conversion difficult is the proliferation of 
data models and levels and styles of DBMS interfaces, in¬ 
ternal data representation, and hardware architecture. In 
this section of the report we will examine the technology 
that has been developed to perform conversions, analyze 
the areas which require new or improved techniques, and 
consider strategies for minimizing the need to convert as 
well as for streamlining those conversions which must be 
undertaken. Over the past six years, research and devel¬ 
opment has primarily centered on the problem of converting 
in non-dynamic environments. The first part of the second 
section surveys tools and technology currently available for 
the conversion of data organization. More recently database 
program conversion—probably the most difficult part of the 
conversion problem—has received attention. It is reviewed 
in the second part of the second section, which considers 
the research directions being taken by current research. The 
final component of the second section analyzes the status of 
the entire technology. The third section explores the factors 
affecting conversion, the approaches for reducing the need 
to convert data and applications programs, and the impact 
of new software and hardware technologies on conversion. 
The fourth section summarizes the trends in conversion 
needs and tools. 


887 



888 


National Computer Conference, 1978 


Components of the conversion process 

When conversion is necessary, users will be required to 
extract data from their source environment and restructure 
them to the form required for the target environment. While 
the extraction and restructuring may themselves be com¬ 
plex, these processes are frequently complicated further by 
the undisciplined nature of the source data. For example, 
data may exist in duplicate or contain numerous errors and 
multiple inconsistencies. The whole process of extracting it 
from its source, “cleaning the data,” restructuring them to 
a desired form, and loading them into the targeted environ¬ 
ment is generally referred to as data conversion or transla¬ 
tion. 

After a data conversion, particularly one involving exten¬ 
sive restructuring, the application programs which process 
the original data may not run correctly against the new data. 
If the amount of restructuring has been small, only a simple 
modification may be required, while extensive rewrite or 
redesign may be required when restructuring has been ex¬ 
tensive. The process of modifying the programs to process 
the restructured data is referred to as application program 
conversion or translation. (In this report we do not consider 
the problem of converting programs not related to a change 
in data structure.) Let us discuss data conversion and ap¬ 
plication program conversion a bit further. 

Concepts of data conversion 

In brief and conceptual terms, the data translation or 
conversion process can be represented diagrammatically in 
Figure 1. As shown in this diagram a data translation system 
generally requires three components: a reader, a restruc- 
turer, and a writer. While the capability of each component 
depends on the individual design of a data translation sys¬ 
tem, the function of the reader in general is to access data 
from its source environment to prepare it for further proc¬ 
essing. The accomplishment of this process unquestionably 
depends on the description of the data. Thus a data descrip¬ 
tion language is needed for this purpose. The writer is the 
functional inverse of the reader, its function being to put the 
transformed data into the target environment. It too requires 
a description of the data structure and shares with the 
reader the need of a data description language. The function 
of the restructurer is quite different from that of the other 
two components. This component in general is responsible 
for the extraction of data from its source or internal forms 
and restructuring it to a desired format or structure. Usually 
a translation description language is required to effect this 
process. 


STORED DATA TRANSLATION STORED DATA 

DESCRIPTION DESCRIPTION DESCRIPTION 



Figure 1—Schematic data conversion process 



Figure 2—Program translation 


In a data conversion system where limited application is 
intended, the three components may not be distinguishable, 
nor is the need of the two languages clear. For example, if 
one wishes to create a conversion system merely to translate 
EBCDIC characters into ASCII, one can create a simple 
system with one component and a simple data description 
language embedded in it. However, if development of a 
broadly applicable data translation system is the goal, it is 
clear that one must have a reader and writer capable of 
accessing and creating data in all kinds of environments and 
a powerful restructurer capable of all manner of manipulat¬ 
ing data and of creating some data as well. Implicit in this 
application is the need for versatile data translation descrip¬ 
tion languages. 

Concepts of database program conversion 

Figure 2 represents a general approach to database pro¬ 
gram conversion or translation. To convert a program it is 
necessary to determine the functions of the program, and its 
semantics. Programmers making assumptions about the 
state of the data may not and currently need not state these 
assumptions explicitly in their programs. Therefore, it will 
usually be necessary to provide more information about the 
semantics of the program than can be extracted from the 
program text and its documentation. Also needed in the 
program conversion process is information about the data 
structure the program originally ran against, the new struc¬ 
ture it must run against, and how the two relate. These 
descriptions could be the same as those used to drive the 
data translation process. The program, the description of its 
semantics, and the description of the data conversion are 
inputs to the program conversion process. It uses them to 
determine the data originally accessed and how to accom¬ 
plish the same access in the new structure, and it produces 
a new program to do this. Currently, the conversion process 
is accomplished through a combination of manual transla¬ 
tion, emulation, and bridge programs. An automatic or semi¬ 
automatic program conversion technology is only in the 
early stages of research. 

CONVERSION TECHNOLOGY 

In the previous section, we discussed the general concepts 
of data and program conversion. In this section we shall 
discuss the technology by which a conversion is achieved. 
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Since most of the conversion results have been achieved 
in data conversion, we focus on this aspect and limit our 
discussion on program conversion to those cases which are 
the result of a data conversion. 

Data conversion technology 

Currently, the common approach to data conversion is to 
develop customized programs for each transfer of data from 
one environment to another. This approach is inherently 
expensive; the programs are developed for use only once 
and their development costs cannot be amortized. Further, 
the reliability is poor as one is likely to make errors as data 
is passed from program to program. For a large data con¬ 
version effort, this latter effect can indeed be quite unman¬ 
ageable, as the tracing to the source of an error in the data 
at a particular point can be extremely complex. An alter¬ 
native is to search for a broader approach for data conver¬ 
sion with a generalized system. We shall now describe such 
systems in more detail. 

Problem discussion 

Data exist in many varied and complex forms; Figure 3, 
an expanded form of the diagram in Figure 1, indicates some 
of the transformations that need to take place in a data 
conversion. This diagram illustrates the capabilities needed 
by the read and write process in a data conversion system. 

The purpose of the unloading of data from its source is to 


reduce the complex physical structure of the source data¬ 
base to a very simple physical structure, namely a sequential 
data stream. The source database contains not only the 
information that interests the user, but also a large amount 
of control information that is specific to a particular system. 
This control information is used by the system for such data 
management functions as overflow chaining, index mainte¬ 
nance, and record blocking. In many systems, when a record 
is to be deleted it is marked or flagged but not actually 
deleted. During the unload transformation, all of this system 
specific information is deleted. Another factor that causes 
complexity in the unloading process is the frequent use of 
pointers in a source system. Pointers are used for two basic 
purposes: (1) to represent relationships that exist between 
record instances and (2) to implement alternative access 
paths to the data. During the unload transformation, the 
second class of pointers may be discarded without loss of 
information. The first class of pointers, however, maintains 
information that must be preserved during the transforma¬ 
tion. 

The purpose of the reformat process is to create a common 
data form of the source data for the restructuring step in the 
conversion process. In the case of a simple sequential file, 
not under DBMS control, it may enter the conversion proc¬ 
ess at this point. Depending on the design of a system, this 
step may involve editing (i.e., encoding or recoding of 
items), limited data extraction, correction, and the like. 
Since the goal in this step is to create a data structure of the 
source data without the system-dependent information, the 
mapping between the input and the output of the reformat 
process can be considered to be generally one-to-one. While 



data data 


description description 

Figure 3—Schematic data conversion process 
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this step looks simple functionally, its actual specification 
and implementation can be quite complex. For example, an 
application program may use the high order bits of a zoned 
decimal number for its own purposes, knowing that these 
bits are not used by the system. The specification of non¬ 
standard item encodings is a difficult problem in data con¬ 
version. 

The load process is the counterpart of the unload process 
and needs no further clarification. Note that, however, the 
use of a common data form provides additional benefits, 
such as easing the portability problem. 

The restructuring process is undoubtedly the most com¬ 
plex process of a generalized data conversion system. The 
languages for this mapping process can be quite different 
(for example, some procedural and other nonprocedural) 
and the models used to represent the data in the conversion 
system are also quite divergent. (For example, some use 
network structure; others use hierarchical structure). More 
will be said on this topic later in this section. 

Let us now turn to discuss the issue of implementation 
briefly. Generally there are two techniques. One can build 
the system using an interpretive approach or the generative 
approach. In the interpretive approach, the action of the 
system will be driven by the descriptions written in the 
system’s languages via the general interpreter implemented 
for the particular system. In the generative approach the 
data and mapping descriptions are fed into the compiler(s) 
which generates a set of customized programs executable 
on a certain machine. The merits of each of these approaches 
will also be discussed later in this section. 

We shall next turn our attention to the tools that have 
been developed for data conversion. We shall first discuss 
the tools currently available and then the research and de¬ 
velopment work in progress. 

Available conversion tools 

Currently all available tools are limited in capability. Be¬ 
cause it is impossible in this short report to provide an 
exhaustive survey of all the vendor developed conversion 
tools, we will highlight the spectrum of capabilities available 
to the user by providing examples from specific vendor 
shops. 

The vendor’s repertoire of conversion tools begins at the 
character encoding level of data conversion with the provi¬ 
sion of hardware/firmware options and continues through 
the software aids for conversion and restructuring of data¬ 
bases. 

Depending on a diversity of conditions, the need to de¬ 
velop software tools varies from vendor to vendor. Probably 
the most prevalent file conversion tool is a COBOL foreign 
file processing aid. This type of facility allows the direct 
reading or writing of a particular class of files such as 
EBCDIC tapes or Honeywell Series 200/2000 files within 
COBOL. Although a relatively widespread facility, its ca¬ 
pabilities are nevertheless limited. For example, some do 
not handle unlabeled tapes, while others cannot process 
mixed mode data types. Aside from the work of Bakkom 
and Behymer, 1 which was aimed toward achieving a general 
conversion bridge with a particular vendor, to our knowl¬ 


edge there are no vendor supported generalized file trans¬ 
lation tools. 

In contrast to the above file translation tools, tools have 
been developed that have their main applications in a data¬ 
base environment. One example of a database conversion 
aid is the I-D-S/II migration aid provided by Honeywell. 
Because of the large volumes of data involved and the fact 
that the user cannot afford to shut down his whole data 
processing shop, a co-existence approach was adopted. The 
first step is to reformat the I-D-S/I database into the I-D-S/ 
II format, making the necessary data type conversions and 
pointer mechanism adaptions. This step allows the data¬ 
base to be processed in the I-D-S/II mode, but not optimally. 
Additional steps in this migration include the generation of 
the additional I-D-S/II pointer fields (I-D-S/II requires 
“Prior & Header” chain pointers, which are allocated in 
step 1 but not filled in) and the restructuring of the I-D-S/II 
(coexistence) to the more sophisticated capabilities of I-D- 
S/1I. 

Some database restructuring tools specific to a particular 
DBMS have been developed by DBMS users. One example 
of this type of tool is REORG, 2 a system developed at Bell 
Laboratories for reorganization of UNI VAC DMS-1100 da¬ 
tabases. REORG provides capabilities for logical and phys¬ 
ical reorganization of a database using a set of commands 
independent of DMS-1100 data management commands. A 
similar capability has been developed at the Allstate Insur¬ 
ance Company. 

In addition to the above, there are also software compa¬ 
nies and vendors who will do a customized conversion task 
on a contractual basis. 

Data conversion prototypes and models 

Over the past seven years a great deal of research on the 
conversion problem has been performed, with the results 
summarized in Figure 4. Projects were initiated at the Uni¬ 
versity of Michigan, the University of Pennsylvania, IBM, 
SDC, and Bell Laboratories, as well as by a task group of 
the CODASYL Systems Committee. In many cases there 
was interaction and cross-fertilization between these groups 
and some consensus on appropriate architectures for data 
conversion was reached. The individual achievements of 
these groups is discussed below: 

The CODASYL Stored-Data Description and Translation 
Task Group 

In 1970 the CODASYL Systems Committee formed a task 
group (originally called the Stored Structure Description 
Language Task Group) to study the problem of data trans¬ 
lation. The group presented their initial investigation of the 
area in the 1970 SIGMOD (then SIGFIDET) annual Work¬ 
shop in Houston. 3 In 1972 the group was reformulated as 
the Stored-Data Description and Translation Task Group 
and a general approach to the development of a detailed 
model for describing data at all levels of implementation was 
presented. 4,5 The most recent work of the group specifies 
the data conversion model and presents an example language 
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CODASYL 

STORAGE STRUCTURE 
DEFINITION LANGUAGE 
TASK GROUP: 


UNIVERSITY OF 
MICHIGAN: 


UNIVERSITY OF 
PENNSYLVANIA: 


IBM RESEARCH, 
SAN JOSE: 


1970 


Specification of the 
data conversion prob¬ 
lem and possible 
approaches to solving 
it 3 


1971 


Model of logical 
structure structure to 
physical structure 
mappings for DBMSs 7,8 


Model and languages for 
Stored-Data description 
and translation descrip¬ 
tion 21 (Moore School) 


1972 


Stored-Data Description 
and Translation Task 
Group: 

Model of generalized 
data conversion archi¬ 
tecture 4,5 


Data Translation Model 5 


1973 


1974 


Restricted prototype 
data translator 
(sequential files) 

11 , 12.83 


Model of restructuring 1 


Restricted prototype 
data translator 22,23 

(Moore School) 


Model and Language for 
data translation 92 
Define 26 


I 


CODASYL 

STORAGE STRUCTURE 
DEFINITION LANGUAGE 
TASK GROUP: 


1975 


1976 


Detailed model of 
stored data 6 


UNIVERSITY OF UNIVERSITY OF 

MICHIGAN: PENNSYLVANIA: 


External test of Penn 
Translator 24 (Moore 

School) 

Scope of project 
broadened to generation 
of business systems 


fer h iM%o T fo a S! t0r 

(network restructuring Dynamic restructuring 70 
and implementation of (Wharton School) 

Translation Descrip- -- 

tion Language 19,57 


IBM RESEARCH, 
SAN JOSE: 

Convert 27 


Implementation of XPRS 2 


Group redirected atten¬ 
tion to program conver¬ 
sion 


Access Path 
Language 


1977 


Specification 


Laboratory of external 
tests of XPRS 


Figure 4 —Summary of the development of data conversion models and 
prototypes 
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Data translation proto¬ 
type 28,30 
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LABORATORIES: 


ADEPT 

Current development of 
data translation 31 
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for describing and translating a wide class of logical and 
physical structures. 6 The stored-data definition language al¬ 
lows data to be described at and distributed to the access 
path, encoding, and device levels. 

The University of Michigan 

The nonprocedural approach to stored-data definition set 
forth by Taylor and Sibley 7,8 provided one of the major 
foundations for the development at the University of Mich¬ 
igan (see Figure 4) of data translators. In concert with Tay¬ 
lor’s language, a model and design for a generalized trans¬ 
lation was initiated by Fry, et al. 9 

The translation model was tested in a prototype imple¬ 
mentation of the Michigan Data Translator in 1972, 10,11 and 
the results of the next implementation, Version I, were re¬ 
ported by Merten and Fry. 12 

In 1974, the work of the Data Translation Project of the 
University of Michigan focused on the database restructur¬ 
ing problem. Navathe and Fry investigated the hierarchical 
restructuring problem by developing several levels of ab¬ 
stractions, ranging from basic restructuring types to low 
level operations. 13 Later, Navathe proposed a methodology 
to accomplish these operations using a relational normal 
form for the internal representation. 14 Version II of the 
Michigan Data Translator was designed to perform hierar¬ 
chical restructuring transformations, but the project did not 
implement it. Instead, the research was directed into the 
complex problem of restructuring network type databases. 
To address this problem Deppe developed a dynamic data 
model—the Relational Interface model—which simultane¬ 
ously allowed a relational and network view of the data¬ 
base. 15 This model formed the basis of the Version IIA 
design and implementation of generalized restructuring ca¬ 
pabilities. 16-18 Another component necessary for the devel¬ 
opment of a restructurer was the formulation of a language 
in which to express the source to target data transforma¬ 
tions. This language, termed Translation Definition Lan¬ 
guage (TDL), evolved through each translator version be¬ 
ginning with a source-to-target data item “equate list” in 
the Version I Translator to the network restructuring spec¬ 
ifications of Version IIA. While the initial version of the 
TDL was quite simplistic, the current version, the Access 
Path Specification Language, 19,20 provides powerful capa¬ 
bilities for transforming network databases. 

The University of Pennsylvania 

Concurrent with the work at the University of Michigan, 
Smith at the University of Pennsylvania (see Figure 4), also 
took a data description approach and developed a stored- 
data definition language (SDDL) for defining storage of data 
on secondary storage devices, and a translation description 
language (TDL) 5,21 and three levels of database structures, 
the logical, storage, and physical, are described using the 
SDDL. In order to describe the source-to-target data map¬ 
pings a first order calculus language was used. Following 
from this work, Ramirez 22,23 implemented a language-driven 
“generative” translator which created PL/I programs to per¬ 
form the conversion. One of the first reports on the utili¬ 
zation of generalization translation tools was provided by 


Winters and Dickey. 24 Using the translator developed by 
Ramirez, they installed it on their system, and applied it to 
conversion IBM 7080 files. 

IBM Research, San Jose 

In 1973 another major data translation research endeavor 
was initiated at the IBM Research Laboratory in San Jose, 
California. Researchers in this project—initially Housel, 
Lum, and Shu, later joined by Ghosh and Taylor—adopted 
the general model as specified in Figure 1 but made several 
innovations. First, in the belief that programmers know well 
the structure of the data in a buffer being passed from a 
DBMS to the application program, the group concentrated 
its effort on designing a data description language appropri¬ 
ate for describing data at this stage. Second, regardless of 
the data model underlying any DBMS, the data structure at 
the time it appears in the buffer of an application program 
will be hierarchical. The general architecture, methodology, 
and languages reflecting these beliefs is reported in Lum et 
al. 25 

In addition, the group in San Jose felt that, while it is 
desirable to have a file with homogeneous record types, it 
is a fact of life that many of today’s data are still in COBOL 
files in which multiple record types frequently exist within 
the same file. As a result the group concentrated on design¬ 
ing a data description language which can describe not only 
hierarchical records (in which a relational structure is a 
special case) but also most of the commonly used sequential 
file structures. This language, DEFINE, is described by 
Housel et al. 26 

The philosophy of restructuring hierarchies is further re¬ 
flected in the development of the translation definition lan¬ 
guage CONVERT, as reported by Shu et al. 27 This language, 
algebraic in structure, consists of a dozen operators, each 
of which restructures one or more hierarchical files into 
another file. The language possesses the capability of se¬ 
lecting records and record components, combining data from 
different files, built-in functions (e.g., SUM and COUNT), 
and the ability to create fields and vary selection on the 
basis of a record’s content (a CASE statement). 

A symmetric process occurs at the output end of the 
translation system. Sequential files are created to match the 
need of the target loading facility. The specification of this 
structure is again made in DEFINE. 

A prototype implementation, originally called EXPRESS 
but renamed XPRS, is reported in Reference 28. 

System Development Corporation 

Another restructuring project reported by Shoshani 29,30 
was performed at The System Development Corporation in 
1974-1975. In order to avoid the complexities of storage 
structure specification (i.e., indexes, pointer chains, in¬ 
verted tables, and the like) they chose to use existing facil¬ 
ities of the systems involved. In particular they advocated 
the use of query and load (generate) facilities of database 
management systems. However, when such facilities do not 
exist, reformatters from the source (e.g., index sequential 
file) to a standard form and from the standard form to same 
output file had to be used. Given that databases can be 
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reformatted to and from a standard form, they concentrated 
on the problem of logical restructuring of hierarchical data 
bases in this form. 

The language used in the above project for specifying the 
restructuring functions (called CDTL—Common Data 
Translation Language) was designed to be conceptually sim¬ 
ple. For the most part it provides functions for specifying a 
mapping from a single field (or combination of fields) of the 
source to a single field of the target. For example, while a 
DIRECT would specify a one-to-one mapping of source 
items to target items, a REPEAT would specify the repeti¬ 
tion of a source item for all instances of a lower level in the 
target hierarchy. In both cases only the source and target 
fields need to be mentioned as parameters. In addition there 
are more global operations, such as the INVERSION op¬ 
erator, which causes parent/dependent record relationships 
to be reversed. The system also supported extensive field 
restructuring operators, where individual field values could 
be manipulated according to prescribed language specifica¬ 
tions. Since most of these operators are local, there is the 
possibility that they could be used in combinations that do 
not make sense globally. Therefore a further component of 
the system was built to perform “semantic analysis,” which 
checks for possible inconsistencies before proceeding to 
generate the target database. 

Bell Laboratories 

The Bell Labs data translation system ADAPT (A DAta 
Parsing and Transformation system), currently under devel¬ 
opment, is a generalized translation system driven by two 
high-level languages. 31 One language is used to describe the 
physical and logical format and structure of the data and to 
provide various tests and computations to be used while 
parsing the source data and generating the target data. The 
second language is used to describe the transformations 
which are to be applied to the source data to produce the 
target data. Extensive validation criteria can be specified to 
apply to the source and target data. 

Two processing paths are available within the ADAPT 
system: a file translation path and a database translation 
path (see Figure 3). A separate path for file translation is 
provided in response to real-world considerations. Many 
types of conversion do not require the capabilities and as¬ 
sociated high overhead involved in using a database trans¬ 
lation path. 

Related work 

Additional research effort is being devoted to the devel¬ 
opment and acceptance of a standard interchange form. An 
interchange form would increase the sharing of databases 
and provide a basis for development of generalized data 
translators. The Energy Research and Development Admin¬ 
istration (ERDA) has been supporting the Interlaboratory 
Working Group for Data Exchange (IWGDE) in effort to 
develop a proposed data interchange form. The proposed 
interchange form 32 has been used by several ERDA labo¬ 
ratories for transporting data between the laboratories. Ad¬ 
ditional work on development of interchange forms has been 


pursued by the Database Systems Research Group at the 
University of Michigan. 33 

Navathe 34 has recently reported a technique for analyzing 
the logical and physical structure of databases with a view 
to facilitating the restructuring specification. Data relation¬ 
ships are divided into identifying and nonidentifying types 
in order to draw an explicit schema diagram. The physical 
implementation of the relationships in the schema diagram 
is represented by means of a schema realization diagram. 
These diagrammatic representations of the source and target 
databases could prove to be very useful to a restructuring 
user. 

Application program conversion 

So far we have concentrated on the data aspects of the 
conversion problem; it is necessary to deal as well with the 
problems of converting the application programs which op¬ 
erate on the databases. Program conversion, in general, may 
be motivated by many different circumstances, such as hard¬ 
ware migration, new processing requirements, or a decision 
to adopt a new programming language. Considerable effort 
has been devoted to special tools such as those to assist 
migration among different vendor’s COBOL compilers, and 
general purpose “decompilers” to have been developed to 
translate assembly language programs to equivalent software 
in a high level language. While progress has been made 
developing special purpose tools for a limited program con¬ 
version situation, little progress has been made in obtaining 
a solution to the general problem of program conversion. 
With this fact in mind, this section focuses on the modifi¬ 
cations to application programs that arise as a consequence 
of data restructuring/conversion. 

Problem statement 

There are three types of database changes which can 
affect application programs: 

• alterations to the database physical structure, for ex¬ 
ample, the format and encoding of data, or the arrange¬ 
ment of items within records. 

• changes to the database logical structure—either 

1. the deletion or addition of access paths to accom¬ 
modate new performance requirements, or 

2. changes to the semantics of data, for example, mod¬ 
ification of defined relationships between record 
types or the addition or deletion of items within 
records 

• migration to a new DBMS, perhaps encompassing a 
data model and/or data manipulation language different 
from the one currently in use. 

The actual impact of these database changes on applica¬ 
tion programs is a function of the amount of data independ¬ 
ence provided by the Database Management Systems. Data 
independence and its relationship to the conversion problem 
are discussed elsewhere. 59 We assume here that data inde¬ 
pendence is not complete and that therefore some degree of 
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program conversion is required in response to database 
schema changes. In fact, whereas most commercial database 
management systems provide application programs with in¬ 
sulation from a variety of modifications to the physical da¬ 
tabase, protection from logical changes—particularly at the 
semantic level—is minimal. Examples of semantic changes 
that are likely to have a profound effect on application pro¬ 
grams include: 

• Changes in relationships between record types, such as 
changing a one-to-many association to a many-to-many 
association or vice versa. 

• Deletion or addition of data items, record types, or 
record relationships. 

• Changing derivable information (“virtual items”) to ex¬ 
plicit information (“actual items”) or vice versa. 

• Changes in integrity, authorization or deletion rules. 

There are various properties of database application pro¬ 
grams that greatly complicate the conversion problem. For 
instance many database management systems do not require 
that the record types of interest (or possibly even the data¬ 
base of interest) be declared at compile time in the program; 
rather these names can be supplied at run time. Conse¬ 
quently at the compile time, incomplete information exists 
about what data the program acts on. Other troublesome 
problems occur when programs implicitly use characteristics 
of the data which have not been explicitly declared (e.g., a 
COBOL program executes a paragraph exactly ten times 
because the programmer knows that a certain repeating 
group only occurs ten times in each record instance). Com¬ 
plexity is introduced whenever a data manipulation language 
is intricately embedded in a host language such as COBOL. 
The interdependence between the semantics of the database 
accesses and the surrounding software greatly complicates 
the program analysis stage of conversion. Because of these 
considerations, substantial research has been devoted to 
alternatives to literal translation of programs. In particular 
some currently operational tools utilize source program em¬ 
ulation or source data emulation at run time to handle the 
problem of incomplete specification of semantics and yet 
still yield the effects of program conversion. 

Current approaches 

In this section, we discuss two main techniques currently 
employed in the industry. These techniques are commonly 
used but unfortunately not documented in the form of pub¬ 
lications. 

DML statement substitution 

The DML statement substitution technique, which can 
be considered an emulation approach, preserves the seman¬ 
tics of the original code by intercepting individual DML 
statements calls at execution time, and substituting new 
DML statement calls which are correct for the new logical 
structure of the database. Two IBM software examples 
which provide this type of conversion methodology are (1) 
the ISAM compatibility interface within VSAM (this allows 


programs using ISAM calls to operate on VSAM database), 
and (2) the BOMP/DBOMP emulation interface to IMS. This 
program conversion approach becomes extremely compli¬ 
cated when the program operates on a complex database 
structure. Such a situation may require the conversion soft¬ 
ware to evaluate each DML operation against the source 
structure to determine status values (e.g., currency) in order 
to perform the equivalent DML operation on the new da¬ 
tabase. Necessary for the generalization of this approach is 
the development of emulation code for the following cases: 
maintain the run time descriptions and tables for both the 
original and new database organizations, intercept all origi¬ 
nal DML calls, and utilize old-new database access path 
mapping description (human input) and rules to dynamically 
determine what set of DML operations on the new database 
are equivalent to each specific operation on the source da¬ 
tabase. 

Although this approach is straightforward in concept, it 
has several drawbacks. The drawbacks can be categorized 
as degraded efficiency and restrictiveness. Efficiency is de¬ 
graded primarily because each source DML statement must 
be mapped into a target emulation program, which uses the 
new DBMS to achieve the same results. The increased over¬ 
head in program size and/or processing requirements can be 
significant. 

The drawback of restrictiveness comes about because the 
emulation approach inhibits the utilization of the increased 
capabilities of the new DBMS and/or data structure through 
the modeling of the old methods. Additionally dependence 
upon the old program semantics limits the sets of permissible 
new data structures must support all of the semantics of the 
source program if the source program is to continue to 
execute in the same manner. It should be noted that the 
rules can be quite complex, even for the limited situation of 
which the data structure changes preserve semantic equiv¬ 
alence. Therefore, in some instances, just the limited task 
of determining if a change in data structure (given no change 
in the data model) will support a set of source programs will 
be an extensive task. 

Bridge program 

The second method in use today is sometimes referred to 
as the Bridge Program Technique. In this technique, the 
source application program’s access requirements are sup¬ 
ported by reconstructing from the target database that por¬ 
tion of the source database needed. Data reconstruction is 
done by means of “bridge programs.” The source program 
is then allowed to operate upon this reconstructed portion 
of the source database to effect the same results that would 
occur if the source database were not modified. Of course, 
a reverse mapping is required to reflect update and each 
simulated source database segment must be prepared before 
it is needed by the application program. 

This approach suffers from the same types of disadvan¬ 
tages inherent in the emulation approach. Efficiency prob¬ 
lems for complex/extensive databases and programs per¬ 
forming extensive data accessing can make this method 
prohibitively expensive for practical utilization. This tech¬ 
nique is generally found as a “specific software package” 
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developed at a computer installation rather than as a stand¬ 
ard vendor supplied package. 

Current research 

Differing from the DML statement substitution and bridge 
program techniques, current research aims toward devel¬ 
oping more generalized tools to automatically or semi-au- 
tomatically modify or rewrite application programs. The 
drawbacks of the existing approaches described above can 
be avoided by rewriting the application programs which 
would take advantage of the new structure and semantics of 
a converted database and by using a general system to do 
the conversion rather than using ad hoc emulation packages 
and bridge programs. 

Research on application program conversion is still in its 
infancy. Consequently, there are very few published papers 
on this subject. A handfull of works are described here in 
the order of the dates of publication. Mehl and Wang 35 
presented a method to intercept and interpret DL/1 state¬ 
ments to account for some order transforms of hierarchical 
structures in the context of the IMS system. Algorithms 
involving command substitution rules for various structural 
changes have been derived to allow the correct execution of 
the old application programs. This approach works only for 
a limited number of order transformation of segments in a 
logical IMS database. Since it is basically an emulation ap¬ 
proach, it has the drawbacks discussed in the previous sec¬ 
tion. 

A paper by Su 36 gives a general model of application 
program conversion as related to database changes resulting 
from a database transformation. An attempt was made to 
identify the tasks required for the automatic or semi-auto¬ 
matic conversion of application programs due to database 
changes. Two main points are stressed in the paper: (1) the 
need for extensive analysis of an application program in¬ 
cluding the analysis of program logic, data variable relations, 
program-subprogram structure, execution profile, etc., and 
(2) the use of database translation operators to determine 
what program transformations are required to account for 
the effects of these operators. The idea of the use of a 
common language to describe the operations of source quer¬ 
ies and the data translation statements is also proposed. 

An approach to the transformation of DBTG-like pro¬ 
grams in response to a database restructuring was proposed 
by Schindler. 37 The approach is based on the concept of 
code templates, which are predefined sequences of host 
language—DML statements (roughly analogous to assembly 
language macros). Application programs can be written as 
nested code templates. The code templates are devised so 
that each one corresponds to an operator in the relational 
algebra. An application program is then mapped into a re¬ 
lational expression, transformations are performed on the 
expression to accommodate the database restructuring, and 
a new program is generated by mapping the transformed 
expression back into code templates. This approach suggests 
that a level of logical data independence may be achieved 
through current programming technology. 

The work by Su and Reynolds 38 studied the problem of 
high-level sublanguage query conversion using the relational 


model with SEQUEL 39 as the sublanguage, DEFINE 26 as 
the data description language and CONVERT 27 as the trans¬ 
lation language. Algorithms for rewriting the source query 
were derived and hand simulated. In this study, query trans¬ 
formation is dictated by the data translation operators which 
have been applied to the source database. The purpose of 
this work was to study the effects of the CONVERT oper¬ 
ators on high-level queries. Only restricted types of SE¬ 
QUEL queries were considered. It is clear from this work 
that a general program conversion system should separate 
the data model and schema dependent factors from the data 
model and schema independent factors; and an abstract rep¬ 
resentation of program semantics and the semantics of data 
translation operators need to be sought so that data conver¬ 
sions at the logic level (especially the type which changes 
the database semantics) and the DBMS level can be at¬ 
tempted. 

Two independent works carried out about the same time 
by Su and Liu 40 and Housel 41 take a more general approach 
to the application program conversion problem. The former 
work is based on the idea that the same data semantics (a 
conceptual model) can be modelled externally by various 
existing data models (relational, hierarchical and network) 
using different schemas. Application programs are mapped 
into an abstract representaton which represents program 
semantics in terms of the primitive operations (called access 
patterns) that can be performed on data entities and asso¬ 
ciations. Transformation rules are then applied on the ab¬ 
stract representation based on the types of changes intro¬ 
duced by the data translation operators. The transformed 
representation is then mapped into another intermediate rep¬ 
resentation (called access path graphs) which is dictated by 
the external model and specific schema used for the target 
database. This representation is then modified by an optimi¬ 
zation component and used for the generation of target 
programs. It is stressed in this work that the semantics of 
both the source and target database be made explicit to the 
conversion system and be used as a basis for application 
program analysis and transformation. The conversion meth¬ 
odology described is for program conversion to account for 
data conversion at the logical level as well as the DBMS 
level. 

The work by Housel is an extension of the work on ap¬ 
plication migration undertaken at the IBM San Jose Labo¬ 
ratory. This work uses a common language for specifying 
the abstract representation of source programs as well as for 
specifying the data translation operations. The language is 
a subset of CONVERT plus some of Codd’s relational op¬ 
erators. The operators of the language are designed to have 
simple semantics and convenient algebraic properties to fa¬ 
cilitate program transformation. They are designed to handle 
data manipulation in a general hierarchical structure called 
a “form” as well as relational tables. In this system, pro¬ 
gram transformation is dictated by the data mapping oper¬ 
ations applied to the source database. It is assumed in the 
proposed model that the inverse of these data mapping op¬ 
erators exists, i.e., the source database can be reconstructed 
from the target database by applying some inverse operators 
on the target database. More precisely, it is assumed that 
M'{T)-S where S is the source database, T is the target 
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database, and M is the mapping function. Thus, program 
conversion is done by substituting the inverse M'(T ) into 
the specification language statements (the abstract represen¬ 
tation of the source program) for each reference to the 
source database. This process is followed by a simplification 
procedure to simplify the resulting statements (the target 
abstract representation of the program). It is pointed out by 
the author that the assumption on the existence of M'(T) 
restricts the scope of the conversion problem handled by 
the proposed approach. 

Presently, the Database Program Conversion Task Group 
(DPCTG) of the CODASYL Systems Committee is investi¬ 
gating the application program conversion problem. The 
group is looking into various aspects of the problems in¬ 
cluding decompilation of COBOL application programs, se¬ 
mantic changes of databases and their effects on application 
programs, program conversion techniques and methodolo¬ 
gies, etc. 

To this date, the work on application program conversion 
is still very much at the research stage and more progress 
has to be made before we can start actual implementation. 
The problems of automatic application program conversion 
are multitudinous and extremely complex. Current research 
indicates that program conversion is possible for some types 
of data conversion, but the complexity of program conver¬ 
sion depends on how drastically the data have been modified. 
Further research needs to be undertaken to determine what 
can be done automatically, what can be done semi-auto- 
matically, and what cannot be done at all. A fully automatic 
tool is hard to achieve. Building semi-automatic systems or 
systems which provide aids for manual conversion would be 
a more realistic goal. 

Current research directions 

The current research has uncovered several problems 
which need to be further investigated before the implemen¬ 
tation of a generalized conversion tool can be attempted. 
The following issues are believed to be important for future 
research: 

Semantic description of database and application 
programs 

Based on the work by Su and Liu 40 and the study of the 
DPCTG group, it is quite clear that a program conversion 
system would need more information about the semantic 
properties of the source and target databases than the infor¬ 
mation provided by the schemas of the existing DBMS. 
Semantic information of the databases is an important 
source for determining the semantics of application pro¬ 
grams which is the real bottleneck of the application program 
conversion problem. Future research needs to be conducted 
to (1) model and describe the semantics of application pro¬ 
gram, (2) study the meaningful semantic changes to data¬ 
bases and their effect on application programs, and (3) derive 
transformation rules for program conversion which account 
for the meaningful changes. Several existing works on da¬ 
tabase semantics 42-46 may provide a good basis for future 
works on this subject. 


Equivalency of source and target programs 

Data conversion may alter the semantic contents of the 
source database. A converted application program may or 
may not perform the identical operation on the target data 
as the source program on the source data. For example, it 
may not retrieve, delete, or update the same data as the 
source program because some records may be deleted and 
data relation may have been changed in data conversion. It 
is not clear at all how we can prove, in general, that a target 
program generated by a conversion system still preserves 
the original intent of the source program. Naturally, if the 
source data can be reconstructed from the target data with¬ 
out losing the original data relations and occurrences, we 
can establish the equivalence relation between the source 
and target programs based on the same effect they have on 
the source and target data. 


Decompilation 

Program conversion via decompilation is a technique 
whereby a database application program is first transformed 
into an operationally equivalent higher order language or an 
abstract representation and then returned to a usable lan¬ 
guage level in a converted form. The transformation to a 
higher order language level is a decompilation process and 
the process of returning the program to a language level 
appropriate to conventional compilers is a compilation pro- 
cesss. The underlying concept is that the decompilation to 
a higher order language can produce a functionally equiva¬ 
lent program that does not contain the DBMS, data model 
and data dependencies that inhibit the conversion process. 
That is, the decompiled program has the same “intent” 
while being unrelated to the changed DBMS environmental 
conditions. The changed environmental conditions should 
be easily incorporated into the program during the process 
of compiling the program back into a form appropriate to 
the new system. 

There is some thought among researchers that this would 
be the preferred method to effect DML/host language pro¬ 
gram conversion. It should avoid many of the efficiency/ 
restriction drawbacks inherent in current automated meth¬ 
ods, while being more cost effective and less error prone 
than current manual methods (e.g., program rewrite). 

One likely disadvantage to this method is that in order to 
use it to convert existing database application programs the 
programs may have to first be manually altered to place 
DML related code in a structured format. This disadvantage 
is to be expected because of the ambiguity inherent in the 
organization of DML/host language programs. However, the 
development of structured programming templates designed 
for DML related code should provide a means for creating 
programs that are convertible by the decompilation method. 
Structured templates might also provide the needed insight 
toward the development/selection of an appropriate high 
level language into which programs can be compiled. Some 
initial concepts of database program templates have been 
proposed by the University of Michigan. 37 
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Conversion aids 

A system which provides assistance to conversion ana¬ 
lysts would seem to be a practical tool and a feasible task. 
Given the information about data changes and semantics of 
the data, a system can be built to analyze application pro¬ 
grams to (1) identify and isolate program segments which 
are affected by the data changes, (2) detect inefficient code 
in the programs, (3) produce a program execution profile 47 
which gives an estimate of the computation time required at 
different parts of the program, and (4) detect, in some cases, 
the program code which depends on the programmer’s as¬ 
sumption of data values, ordering of records, record or file 
size, etc. The data obtained in 1 together with some on-line 
editing and debugging aids would speed up the manual con¬ 
version process. The data obtained in 2 and 3 would be 
useful for producing more efficient target programs and the 
data obtained in 4 would help the conversion analyst to 
eliminate the “implicit semantics” in programs which makes 
the program conversion task (manual or automatic) ex¬ 
tremely difficult. To assist the conversion analysts in iden¬ 
tifying ramifications of changes to programs a more com¬ 
plete cross-referencing than that usually produced by 
today’s compilers can be extremely helpful. An example of 
such a product is the Data Correlation and Documentation 
system produced by PSI-TRAN Corporation. One tech¬ 
nique, sometimes used during a conversion process that has 
been initiated by a database structure change, is to alter the 
names of effected database items in the DDL only and use 
errors generated by the compiler to locate program segments 
which need to be changed. A more complete cross-refer¬ 
encing system would be a much better tool, if it were avail¬ 
able. 

Optimization of target program 

As the result of data conversion, multiple access paths to 
the same data may occur. This is because redundant data 
may be introduced or new access paths may be added in the 
course of data conversion. In this situation, a conversion 
system will have the choice of selecting a path to generate 
the target program. The efficiency of the program during 
execution time may depend on the selection of optimized 
access path during program conversion. Also, for reasons 
of achieving generality, some program conversion tech¬ 
niques proposed 40 * 41 convert small segments of programs or 
the equivalent of DML statements separately. It is necessary 
to do a global optimization or simplification to improve the 
converted program. Techniques for program optimization 
related to program conversion need to be investigated. 

Analysis of prototype conversion systems 

This section analyzes the state of the art of generalized 
data conversion systems. It summarizes what has been 
shown to be technically feasible and points out what has 
been learned in the various prototypes. The prototypes have 
yielded encouraging results, but some weak points have also 
emerged. Later sections list some questions that remain to 
be answered and comment on additional features that will 
be necessary to enhance usability and analyze some imple¬ 


mentation issues which can affect the cases where a gener¬ 
alized conversion system can be applied. 

Where do we stand 

The prototype systems described earlier have been used 
in a few conversions. While some of these tests were made 
on “toy files”, a few of the tests involved data volumes 
from which realistic performance estimates can be extrap¬ 
olated. This section will summarize the major tests that were 
done with each of the prototypes. 

The Penn Translator 

The translator developed by Ramirez at the University of 
Pennsylvania 22,23 processes single sequential files to produce 
single sequential target files. Facilities exist for redefining 
the structure of source file records, reformatting and con¬ 
verting accordingly. Conversion of the file can be done se¬ 
lectively using user-defined selection criteria. Block size, 
record size, and character code set can be changed, and 
some useful data manipulation can be included. 

The translator was used in several test runs on an IBM/ 
370 Model 165. The DDL to generated PL/I code expansion 
ratio was 1:4, so coding time was reduced. 

A further test of the Penn Translator was conducted by 
Winters and Dickey. 24 An experiment was conducted com¬ 
paring a conventional conversion effort against the Penn 
Translator (slightly modified). Two source files stored on 
IBM 1301 disks under a system written for the IBM 7080 
using the AUTOCODER language were converted to two 
target files suitable for loading into IMS/VS. Much of the 
data was modified from one internal coding scheme to an¬ 
other. The conversion required multiple source files to mul¬ 
tiple target files. 

The conventional conversion took several months versus 
five months for the generalized approach, a productivity 
improvement of roughly thirty percent. Time for adapting 
the translator, learning the DDL, and adapting to a new 
operating system is included in the five month figure. With¬ 
out these, an estimate of three months was made for the 
conversion using the generalized approach. 

The SDC translator 

The translator described in References 29 and 30 was 
implemented during 1975-1976. The translator could handle 
single, hierarchical files from any of three local systems— 
TDMS, a hierarchical system which fully inverts files; DS/ 
2, a system which partially inverts files; and ORBIT, a 
bibliographic system which maintains keys and abstracts. 
Databases were converted from TDMS to ORBIT, from 
TDMS to DS/2, and vice versa, and from sequential files to 
ORBIT. TDMS files were unloaded using an unload utility. 
Target databases were loaded by target system load utilities. 

The total effort for design and implementation was about 
three man-years. The system was implemented in assembly 
language on an IBM/370 Model 168, and occupied about 40 
K-bytes, not including buffer space which could be varied. 
The largest file tested was on the order of five million char- 
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acters and the total conversion time was about 1 minute of 
CPU time per 2.5 megabytes of data. 

The work was discontinued in 1976. 

The Honeywell translator 

The prototype file translator developed at Honeywell by 
Bakkom and Behymer 1 performed file conversions (one file 
to one file) among files from IBM, Honeywell 6000, Honey¬ 
well 2000, Honeywell 8200 sequential and indexed sequential 
files. Data types of fields could be changed as well as field 
justification and alignment. New fields could be added to a 
record and fields could be permuted within a record. File 
record format (fixed, variable, blocked, etc.) could be 
changed and a compare utility was available for checking 
the consistency of files with different field organizations and 
encodings. Tests of up to 10,000 records were run. Perform¬ 
ance of 15 milliseconds per record was typical (Honeywell 
Series 6000 Model 6080 computer). The prototype has been 
used in a conversion/benchmark environment but has not 
been offered commercially. 

The Michigan data translator 

Version IIB, Release 1.1 of The Michigan Translator was 
completed for the Defense Communications Agency in Oc¬ 
tober 1977. 48 It offers complete conversion/restructuring fa¬ 
cilities for users of Honeywell sequential, ISP, or I-D-S/I 
files. Up to five source databases of any type may be 
merged, restructured or otherwise reorganized into as many 
as five target databases, all within a single translation. Da¬ 
tabase description is accomplished by minor extensions to 
existing I-D-S DDL statements. Restructuring specification 
is easily indicated via a high level language. Tests performed 
to date included a conversion of a 150,000 record I-D-S/I 
database with a total elapsed time of 24 hours (500 millisec¬ 
onds per record). A given translation can be broken off at 
any point to permit efficient utilization of limited resources 
and also protect against system failures. The user is provided 
with the capability of monitoring translation progress in real 
time. 

XPRS 

Test cases with the XPRS system have focussed on func¬ 
tionally duplicating real conversions which had been done 
previously by conventional methods. Several cases have 
been programmed. In each case at least two input files were 
involved. Generally there was a requirement to select some 
instances from one file, match with instances in another file, 
eliminate some redundant or unwanted data, and build up 
a new hierarchical structure in the output. In several cases 
there was a need for conditional actions based on flags 
within the data. In all cases, the XPRS languages were found 
to be functionally adequate to replicate the conversion. A 
productivity gain of at least fifty percent in total analysis, 
coding, and debugging time was achieved. Test runs were 
conducted on several thousand records. Performance was 
deemed adequate in that XPRS can restructure data at least 
as fast as it can be delivered from direct access storage. No 


detailed performance comparisons were made comparing 
XPRS-generated programs with custom written programs. 

Questions remaining to be answered 

Given that several prototype data translation systems are 
operational in a laboratory environment, there is a little 
question concerning the technical feasibility of building gen¬ 
eralized systems. The remaining questions pertain to the use 
of generalized systems in “real world” data conversions 
involving a wide variety of data structures, very large data 
volumes, and significant numbers of people. Three major 
questions to be resolved are: 

1. Are the generalized systems functionally complete 
enough to be used in real conversions, and if not, what 
will it take to make them functionally complete? 

2. Can the people involved in data conversions use the 
languages? What additional features are necessary to 
enhance usability? 

3. Overall, what is the productivity gain available with 
the generalized approach? 

Within the next year, prototype systems will be exercised 
on a variety of real-world problems in data translation, and 
concrete answers to these questions should be available. 
The systems being further tested for cost-effectiveness are 
the Michigan Data Translator, the IBM XPRS system, and 
the Bell Laboratories ADAPT system. 

To date, preliminary results have been promising. A sig¬ 
nificant sample size on which to do analysis of productivity 
gain should be available at the end of the year of testing. 

A number of factors must be taken into account in meas¬ 
uring the cost-effectiveness of the generalized data translator 
versus the conventional conversion approach. These factors 
include: 

—ease of learning and using the higher level languages 
which drive the generalized translators; 

—availability of functional capability to accomplish real- 
world data conversion applications within the general¬ 
ized translators; 

—overall machine efficiency; 

—correctness of results from the conversion; 

—ability to respond in timely fashion to changes in con¬ 
version requirements (conversion program “mainte¬ 
nance”); 

—debugging costs; 

—ability to provide “bridge back” of converted data to 
old applications; 

—ability to provide verification of correctness of data 
conversion; 

—capabilities for detection and control of data errors. 

The languages used to drive generalized data translators 
are high-level and non-procedural; they provide a “user- 
friendly” interface to the translators. Since the languages 
are high-level, programs written in them have a better 
chance of being correct. Experience to date with DEFINE 
and CONVERT, the languages which drive XPRS, has 
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shown that users can learn these languages within a week; 
it has also shown that some practice is necessary before 
users start thinking about their conversion problem in non¬ 
procedural rather than procedural terms. 

In early test cases, the languages which drive generalized 
data translators have been found to be functionally adequate 
for many common cases. In those cases where a feature is 
lacking, a “user hook” facility is often provided. However, 
forcing a user to revert to a programming language “hook” 
defeats the purpose of the high level approach, and inter¬ 
facing the hook to the system requires at least some knowl¬ 
edge of system interfaces. Thus it is important that the high 
level languages cover the vast majority of the cases in order 
to succeed; otherwise, users will perceive little difference 
over conventional approaches. 

Facilities for detecting and controlling data errors in the ,» 
generalized systems are very important, and most of the pro¬ 
totypes do not yet do a complete job in this area. However, 
the generalized packages offer an opportunity for general¬ 
ized, high-level methods for dealing with data errors during 
conversion, and it could well be that once these error pack¬ 
ages are developed, they will contribute to even larger pro¬ 
ductivity gains than have been experienced to date. 

The high-level language approach to driving generalized 
translators should provide the ability to respond to changes 
in conversion requirements with relative ease. Since large 
conversions often take one or more years, it is not unusual 
for the target database design to change or for new require¬ 
ments to be placed on the conversion system. In other 
words, in a large conversion effort, the programs are not as 
“one shot” as is commonly believed. In large conversions, 
the savings in conversion program maintenance could be 
significant. 

Generalized systems can also be used to map target data 
back to the old source data form, assuming the original 
conversion was information-preserving. This capability pro¬ 
vides a means for verifying the correctness of the data con¬ 
version. In addition, this capability can be used as a “bridge 
back” to allow users to continue to run programs which 
have not yet been converted against the data in the old 
format. Using a generalized system in this way allows 
phased conversion of programs without impacting user 
needs during the conversion period. 

In an environment where a generalized translator is used 
regularly as a tool for conversion, costs associated with the 
debugging phase should be decreased through the use of 
common software modules. It is unusual in the conventional 
approach for common conversion modules to be developed. 
Thus each new conversion system requires debugging. 

Usability 

The usability of generalized data translation systems must 
also be evaluated. Experience to date indicates that the 
languages are easy to learn and use. However, it would be 
wrong to think that these prototypes are mature software 
products or that they can be used in all conversions. This 
section discusses some of the unanswered questions with 
respect to usability of the current data conversion systems. 


One question concerns the level of users of the generalized 
languages. Current prototypes have been used by applica¬ 
tion specialists and/or members of a database support group. 
The systems have not yet been used by programmers, and 
the question remains whether programmers (as opposed to 
more senior application specialists and analysts) will be able 
to use the systems productively. There is no negative data 
on this point; the systems have not been used widely 
enough. 

At present, all the systems require a user to describe 
explicitly the source data to be accessed by the read step 
using a special data description language. These data de¬ 
scription languages are generally easy to learn and use; they 
resemble statements in the COBOL Data Division. How¬ 
ever, the writing of the description is a manual process 
which can be tedious because a person may have to describe 
a file with hundreds of fields. Ideally, a data conversion 
system should be able to make use of an existing data de¬ 
scription, such as those existing in a data dictionary or a 
system COBOL macro library. As evidenced by the Mich¬ 
igan Data Translator, 19 it is reasonable to expect that such 
an interface will be available as data conversion systems 
evolve. It should be pointed out, however, that a data dic¬ 
tionary or COBOL macro library link may not necessarily 
solve the problem. Data in current systems is not always 
fully enough defined to be converted. This is especially true 
with non-database files. In these files, data definition often 
is embedded in the record structures of the programs, and 
a full definition depends on a knowledge of the procedural 
program logic. Even with existing databases, some fields 
and associations may not be fully defined within the system 
database description. Thus, the user can expect a certain 
amount of manual effort in developing data definitions. If 
existing documentation is incomplete, this can be a time 
consuming task, though it probably must be done regardless 
of whether a generalized package is used or not. 

Another area where a user may have to expend effort is 
in the unload step of the data conversion process. The data 
description languages used to drive the read step have a 
limited ability to deal with data at a level close to the hard¬ 
ware (e.g., pointers, storage allocation bit maps, etc.). It is 
generally assumed that a system utility program can be used 
to unload source data and remove the more complex internal 
structures. Another alternative is to run the read step on top 
of an existing access method or database management sys¬ 
tem with the accessing software removing the more com¬ 
plex, machine dependent structures. These approaches are 
acceptable in a great many environments, including most 
COBOL environments, but there may be cases where 
neither approach will work. For example, a load/unload 
utility may not exist, or a file with embedded pointers which 
was accessed directly by an assembly language program 
might not be under the control of an access method. For 
these cases, the user is faced with complexity during the 
unload step. The complexity associated with accessing the 
data would appear to be a factor for either the conventional 
methods or for the generalized approach. However, in cases 
such as those above, some special purpose software may 
have to be developed. It should be noted that some research 6 
has examined the difficulty of extending data description 
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languages to deal directly with these more complex cases. 
The conclusion is that providing the data description lan¬ 
guage with capabilities to deal with more complex data 
structures greatly complicates the implementation and has 
an adverse affect on usability. Thus, special purpose unload 
programs will continue to be required to deal with some 
files. 


Analysis of architectures 

This section discusses some of the different approaches 
that have been taken in implementing the prototype data 
conversion systems. The objective is to analyze some of the 
performance and usability issues raised by the prototypes. 

Two approaches have been used in the prototypes—gen¬ 
erative approach in the Penn Translator and XPRS, and an 
interpretive approach in the Michigan Data Translator. In 
the generative approach, a description of the input files, 
output files, and restructuring operations is fed to a program 
generator. From these descriptions, special purpose pro¬ 
grams are generated to accomplish the described conver¬ 
sion. In both the Penn Translator and XPRS, PL/I is the 
target language for the generator. The generated PL/I pro¬ 
grams are then compiled and run. In the interpretive ap¬ 
proach, tables are built from the data and/or restructuring 
description. These tables are then interpreted to carry out 
the data conversion. 

In data conversion systems, as in other software, an im¬ 
plementation based on interpretation can be expected to run 
considerably more slowly than one based on generation and 
compilation. Initial experience with prototype data transla¬ 
tors has shown that there is much repetitive work, strategies 
for which can be decided at program compilation/generation 
time. Also, there is a good deal of low level data handling, 
such as item type conversions. Thus, those implementations 
based largely on an interpretive approach run more slowly, 
and the ability to vary bindings at run time does not appear 
to be necessary. Interpretation was chosen in the prototypes 
for ease of implementation, and in the future it can be ex¬ 
pected that a compilation-based approach or a mixture of 
compilation with interpretation will be the dominant imple¬ 
mentation architecture. However, for medium scale data¬ 
bases, the machine requirements of the interpretive data 
conversion prototypes are not unreasonable, and overall 
productivity gains are still possible. 

Performance measurements with conversion systems 
based on the generative approach indicate that generalized 
systems can be quite competitive with customized programs. 
In one case, the program generated by the data conversion 
system ran slightly faster than a “customized” program 
which had been written to do the same job. However, this 
example could well be the exception and it would be naive 
to expect this in general. The reason generalized packages 
can be competitive is that they often have internal algorithms 
which can plan access strategies to minimize I/O transfer 
and/or multiple passes over the source data. “Customized” 
conversion programs written in a conventional programming 


language often are not carefully optimized, since the expec¬ 
tation is that the programs will be discarded when the con¬ 
version is done. 

A second architectural difference involves the use of an 
underlying DBMS or not. In both the Penn Translator and 
XPRS, the generated PL/I program, then executing, ac¬ 
cesses sequential files, performs the restructuring, and 
writes sequential files. On the other hand, the Michigan Data 
Translator functions as an application program running on 
a network structured database management system. Thus 
the interpreter makes calls to the underlying DBMS to re¬ 
trieve data during restructuring and puts restructured data 
into the new database. 

The two approaches offer different tradeoffs. For exam¬ 
ple, the Michigan Data Translator can make use of the ex¬ 
isting extraction capabilities of a DBMS and perform partial 
translations easily. In addition, since it operates directly 
within the network data model, a user does not have to think 
of “unloading” data to a file model and then reloading it 
back; rather, the user describes a network to network re¬ 
structuring much more directly. 

On the other hand, when converting non-database data to 
a database, the use of an underlying DBMS as part of a data 
translator implies a second order data conversion problem— 
the non-database data must be converted into the DBMS of 
the data conversion system, which may or may not be dif¬ 
ficult. It can be difficult, for example, when the data model 
of the data being converted differs significantly from the 
data model of the DBMS upon which the conversion system 
is based. Also, the use of an underlying DBMS may also 
require more on-line storage, whereas the file oriented con¬ 
version systems can be made to run tape-to-tape. This can 
be important in very large database conversions. 

In the future, one can expect that data conversion systems 
will offer a variety of interfaces to accommodate various 
kinds of conversion situations. For example, it is possible 
to interface the “file-oriented” conversion systems to run 
as application programs on top of existing database man¬ 
agement systems. It is also possible to develop “reader 
programs” to load non-database data into conversion sys¬ 
tems based on a DBMS. In addition, more automated inter¬ 
faces to data dictionary packages can be expected in order 
to improve usability and obviate the need for multiple data 
definitions. 

One possible performance problem with generalized con¬ 
version systems lies in the unload phase. For reasons of 
usability, generalized conversion systems usually rely on an 
unload utility program to access the source data, thus iso¬ 
lating the conversion package from highly system specific 
data. A potential problem with this approach is that the 
unload package may not make good use of existing access 
paths or may tend to access the source data in a fashion 
which assumes that the data has recently been reorganized 
(with respect to overflow areas, etc.). In cases where the 
data is badly disorganized, a customized unload program 
which accessed the data at a lower level might run consid¬ 
erably faster, and for very large databases might be the only 
feasible way to unload the data. It is not clear how common 
this case is, and one can usually make the argument that the 
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“special” unload software could be interfaced to the gen¬ 
eralized package. However, from a practical standpoint, the 
unloading phase on a very large, badly disorganized data¬ 
base is a performance unknown, and more sophisticated 
unload utilities may have to be developed as part of the 
generalized packages. 

Summary 

Detailed performance and productivity figures for major 
conversions should be available in about one year. Expec¬ 
tations are that machine efficiency of the generalized pack¬ 
ages based on a generation/compilation approach will be 
acceptable (no worse than a factor of 2) when compared 
with conventional conversion programs. Additional en¬ 
hancements to improve usability can be expected, especially 
in the areas of data error detection and control and interfaces 
to data dictionary software. If the savings in conversion 
program analysis and coding times—often fifty percent or 
more—are confirmed, then the generalized conversion sys¬ 
tems will be ready for extensive use. 

OTHER FACTORS AFFECTING CONVERSION 

In this section we look at the conversion problem from 
two aspects. First we address the question—What can we 
do today to lessen the impact of a future conversion. Sec¬ 
ond, we look to the future to see what effects future tech¬ 
nology and standards will have on the conversion process. 

Lessening the conversion effort 

In order to identify guidelines for both reducing the need 
for conversion and for simplifying conversions which are 
required, it is necessary to consider the entire application 
software development cycle. This is because poor applica¬ 
tion design, poor logical database design, inadequate use of 
and inappropriate DBMS selection each could lead to an 
environment which may prematurely require an application 
upgrade or redesign. This redesign could, in many cases, 
require a major database conversion effort. 

The set of guidelines specified below is not intended as a 
panacea. Instead, it is meant to make designers aware of 
strategies which make intelligent use of current technology. 
It is doubtful that all conversions could be avoided if a 
project adhered strictly to these proposed guidelines. How¬ 
ever, adherence to the principles set forth by these guide¬ 
lines could certainly reduce the probability of conversion, 
and more importantly, simplify the conversions that are 
required. 

With respect to application design and implementation, 
the more the application is shielded from system software 
and hardware implementation details, the easier it becomes 
for a conversion to take place. For example, a good se¬ 
quential access method hides the difference between tapes, 
disks, and drums from the application programs which use 
the access method. 


The logical database design should be specified with a 
clear understanding of the information environment. A good 
logical database design reduces the need to restructure be¬ 
cause it actually models the environment it is meant to 
serve. Introduction of data dependencies in the data struc¬ 
ture should, if possible, be kept to a minimum. An analysis 
of the tradeoffs between system performance and likelihood 
of conversion should definitely be made. 

Selecting the wrong or non-optimal database management 
system, given the application requirements, is also a key 
problem which can lead to unnecessary and large conversion 
efforts. The prospective user of a DBMS should, for ex¬ 
ample, carefully evaluate the data independence character¬ 
istics of a proposed DBMS. 

The underlying principle of the guidelines which follow is 
that decisions can be made at the system design and imple¬ 
mentation stages which are crucial to the stability of the 
applications. 


' Application design guidelines 

Requirements analysis 

Many of the decisions affecting the long-term effective¬ 
ness of the application system (database design as well as 
application programs) are made during the requirements 
analysis stage of system development. Questions such as 
what are the functional requirements of the application, who 
the database is to serve, how the database is to be used, 
what are the possible future uses of the data, and what are 
the performance constraints of the application answered at 
this stage. It is essential that the designer understand the 
information environment as much as possible at the outset 
in order to lessen the probability that frequent conversions 
will be necessary. 

Requirements analysis should focus on information needs 
and should minimize constraints being imposed by the phys¬ 
ical environment. Imposing physical constraints at this junc¬ 
tion is dangerous since it can distort the designer’s view of 
the true objectives of the application system. The influence 
of the physical environment should be considered secon¬ 
darily, in order that the designer be fully aware of the resulting 
compromises to the logical requirements. This is not in¬ 
tended to imply that consideration of the physical environ¬ 
ment is unimportant. Indeed, if the physical environment is 
ignored the effect could be development of a set of require¬ 
ments that are impossible to meet within existing physical 
and cost constraints. 

Program design guidelines 

There are three underlying principles motivating this dis¬ 
cussion of application program design. They are: 

• design for maintainability 

• design for the application 

• data independence 
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Keeping sight of all of these during the design of the appli¬ 
cation program will lessen conversion effects by rendering 
the application as free as possible from physical considera¬ 
tions. 

Designing for maintainability implies that the application 
should be written in a high-level language with a syntax that 
permits good program structure. Structured programming 
techniques such as top-down program design and implemen¬ 
tation should be used throughout. The system should be 
modular with relatively small, functionally oriented pro¬ 
grams. The programs should all be well commented and 
organized for readability. Design reviews and program walk¬ 
throughs also help to expose errors in the overall design and 
“holes” in the application logic at an early stage. It has 
been well documented that these steps help in making pro¬ 
gram modifications a much easier task. 

One error which is often made in designing programs in 
a DBMS environment is to let the capabilities of the DBMS 
drive the design rather than the application. This design 
error can yield programs which are unnecessarily dependent 
upon the features of a specific DBMS. For example, in 
System 2000 one can use a tree to represent a many-to-many 
relationship instead of using the LINK feature. The parent/ 
child dichotomy that results is an efficient but arbitrary 
contrivance that cannot easily be undone later on. The key 
principle here is to concentrate on what results are desired 
rather than on the implementation details of achieving these 
results. Simplicity and generalization of the design will pro¬ 
vide a very high level of interface to the application pro¬ 
grammer which will, in turn, minimize the total amount of 
software, provide the greatest degree of portability, main¬ 
tainability, device independence, and data independence. 

Of extreme importance in program design is the notion of 
data independence, i.e., insulating the application program 
from the way the data is physically stored. 

Layered design 

In the area of application design, the motivating factor for 
mitigating the effects of a conversion is to insulate logical 
operations from physical operations. One of the concepts 
applied to achieve this is layered design. That is, designing 
the application as a series of layers, each of which com¬ 
municates with the system at a different level of abstraction. 
One can visualize this as an “onion,” with hardware as its 
core and layers of successively more sophisticated software 
at the outer layers. The user interacts with the outermost 
skin of the onion, at the highest level of abstraction. 

If application programs are written at the outermost layers 
of the onion, then these programs are smaller, easier to 
understand, and therefore easier to modify or convert than 
programs written at lower layers. For example, introduction 
of a new mainframe will require conversion of the software 
which references the particulars of the mainframe. How¬ 
ever, since the layers are constructed so that physical ma¬ 
chine and device independence is realized above some level, 
only the software below that level is subject to modification. 
To the extent that application programs stay at the outer¬ 


most layers (i.e., above the critical layer) reduced conver¬ 
sion effects can be achieved. 

We can thus summarize the goal of program design as 
follows: 

• to provide the highest possible application interface to 
the program 

• to maximize program independence from the charac¬ 
teristics of the mainframe, peripherals, and database 
organization 

• to maximize portability of the application program 
through the use of high-level languages 

• to maintain a clean program/data interface 


Programming techniques 

The previous sections of this chapter have focused on the 
design decisions which should be made to alleviate the con¬ 
version problem. However, regardless of how noble these 
goals are, poor implementation decisions can go a long way 
towards diminishing the returns of a good design. Equally 
important to intelligent design is a set of programming tech¬ 
niques and standards which prohibit programmers from in¬ 
troducing dependencies in code. For example, a “clever” 
programmer may introduce a word size dependency in a 
program by using right and left shifts to effect multiplication 
and division. Of course, there are no hard and fast safe¬ 
guards against using tricky coding techniques; an effort must 
be made to make the programmer conscious of the conse¬ 
quences of this kind of coding. In particular, a programmer 
should not be allowed to jump across layers of the onion, 
such as to use an access method to directly read or write 
databases. 

Database design 

Perhaps the most costly mistake a designer can make is 
an error in the database design because it has a direct effect 
on the information that is derivable and the application pro¬ 
grams that are created. Incorrect or unanticipated require¬ 
ments can lead either to information deficient databases or 
overly complex and general design. An inadequate logical 
design has the potential for complex user interfaces or ex¬ 
tremely long access times. A poor physical design can lead 
to high maintenance and performance costs. Unfortunately, 
database design is still an art at the present time. Two 
surveys report the results in the area to date. Novak and 
Fry 49 survey the current logical database design methodol¬ 
ogy and Chen and Yao 50 review database design in general. 
The work of Bubenko 51 in the development of the CADIS 
system and the abstraction and generalization techniques of 
Smith and Smith 46,52 show promise. 

An accurate logical design can still be unnecessarily data 
dependent. Dependencies are inadvertantly or deliberately 
introduced in the interest of improving system performance. 
In essence, “purity” is compromised to gain processing 
efficiencies. Since optimization is a worthwhile goal, insist- 
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ing on absolute purity may be unreasonable. However, the 
database designer should at least be aware of contrivances 
and therefore be in a position to evaluate the relative effects 
a design decision may have. Designers should become sen¬ 
sitive to their decisions by asking: “How will the data model 
be affected by a future change in performance requirements? 
Have I done a reasonable job in insulating applications from 
data structure elements that are motivated strictly by per¬ 
formance considerations?” 

Some examples of induced data dependencies in logical 
database design which may impact upon conversion are: 

• The use of owner-coupled sets in DBTG to implement 
performance-oriented index structures or orderings on 
records. 

• Storing physical pointers (or database keys) in an in¬ 
formation field of a record. 

• Combining segment types (in DL/1) to reduce the 
amount of I/O required to traverse a database. 

DBMS utilization and selection 

Selection of a DBMS can have a major impact on con¬ 
version requirements. Of importance in evaluating a DBMS 
is to consider products exhibiting the highest level user 
interface. 

A high level DBMS is characterized by both a powerful 
set of functions and a high degree of data independence from 
the point of view of the application. With respect to func¬ 
tions, that is, the DML, the distinction between “high level” 
and “low level” has traditionally centered on whether the 
DBMS provides user operations on sets of records (select, 
retrieve, update, or summarize all the records or tuples 
which satisfy some conditions) or whether one is restricted 
to record-at-a-time processing (“navigation”). The DBMS 
with the “high-level” set operation approach is significantly 
more desirable than the navigational record-by-record ap¬ 
proach . 

DBMS prospects should evaluate the data independence 
characteristics of a proposed product. Systems are preferred 
which support an “external schema” or “subschema” fea¬ 
ture which permits the record image in the application pro¬ 
gram (the user work area) to differ significantly from the 
database format. However, the subschema concept is only 
one aspect of data independence. In general, it is necessary 
to determine in what ways and to what extent the application 
interface is insulated from performance or internal format 
options. For instance, will programs have to be modified if: 

• a decision is made to add or delete an index? 

• the amount of space allocated to an item is increased 
or decreased? 

• chains are replaced by pointer arrays? 

Other conversion related questions about DBMS product 
include the following: 

• Are there adequate performance and formatting alter¬ 


natives? Are there too many (i.e., unproductive or in¬ 
comprehensible) tuning options? Are there adequate 
performance measurement techniques and tools to 
guide the exercise of these choices? 

• Does the system automatically convert a populated da¬ 
tabase when a new format option is selected? 

• Aside from tuning, does the DBMS gracefully accom¬ 
modate at least simple external changes such as adding 
or deleting a record or item type? 

• Are there other useful high level facilities associated 
with or based on the DBMS, such as a report writer, 
query processor, data dictionary, transaction monitor, 
accounting system, payroll system, etc.? 

• Is there a utility for translating the database into an 
“interchange form”, i.e., a machine independent, serial 
stream of characters? 

• Is the vendor committed to maintaining the product 
across new operating system and hardware releases/ 
upgrades? Conversely, is the vendor prepared to sup¬ 
port the product in order of released versions of the 
operating system, so the user will not be forced to 
upgrade? 

• What hardware environments are currently supported 
and what is the vendor’s policy regarding conversion to 
another manufacturer’s mainframe? 

• What programming language interfaces are available? 
Can the same DBMS features be used if there is a 
migration, say, from COBOL to PL/1? 

• How intelligent is the system’s technique for organizing 
data on the media? Specifically, will performance de¬ 
teriorate at an inordinate rate as updating proceeds? 
How often will reorganization (cleanup) be required? 
does the DBMS have a built-in reorganization utility? 
How does the user determine the optimal time to re¬ 
organize? 

• Are the language facilities and data modeling facilities 
of DBMS adequate for the anticipated long term re¬ 
quirements of the enterprise? What is the risk of having 
to convert to a new DBMS? 

• Likewise, are the performance characteristics and in¬ 
ternal storage structure limitations adequate to meet the 
long term requirements (response times, database sizes) 
of the enterprise? 

• Are there facilities to assist the user in converting data 
from a non-DBMS environment or from another 
DBMS? For instance, can a database be loaded from 
one or more user defined files? 

Impact of future technologies and standards on conversion 

In this section we discuss trends in computer hardware 
technologies, DBMS software directions, and standards de¬ 
velopment, and consider their impact on data and program 
conversion. Our intention is to make the reader aware of 
what to expect in terms of conversion problems rather than 
give a complete assessment of future technologies. There¬ 
fore, we discuss only technologies and standards that will 
impact conversion problems. 

The first three parts discuss the areas of hardware, soft- 
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ware, and standards and their impact on conversion in some 
detail. The last part summarizes the major points of our 
assessment without going into detailed reasoning. 

Hardware and architectural technologies 

The cost and performance of processor logic and memory 
continue to improve at a fast rate. As a result, overhead 
costs are more acceptable, especially when such costs save 
people’s time and work, and provide user oriented functions 
that do not require a computer expert. In particular, one can 
now think about using generalized conversion tools not only 
when it is required as a result of hardware or software 
changes, but also as a result of a changing application that 
requires a new more efficient database organization. What 
could have been a prohibitive cost for a database conversion 
in the past, may not be a major factor in the future. 

At the same time, the cost/performance improvement con¬ 
tributes to the proliferation of databases and therefore ac¬ 
centuates the need of generalized conversion tools. The 
most cost effective is the process of accessing and main¬ 
taining data, the more data is collected on computers. Im¬ 
provements in hardware (as well as software) technologies 
create more need for data and program conversion. In ad¬ 
dition, the emergence of new technologies, such as com¬ 
munication networks, add another level of sophistication to 
the way that data can be organized and used. Distributed 
databases, where multiple databases (or subsets of data¬ 
bases) may reside on different machines, require tools for 
the integration and the correlation of data. Invariably, data 
will need to move from system to system dynamically, pos¬ 
sibly moving between different hardware/software systems. 
In this environment generalized tools for dynamic conver¬ 
sion will become a necessity. 

In recent years two promising approaches to data man¬ 
agement hardware technologies have been pursued. One is 
the specialized data management machine and the other is 
the backend data management machine. As will be explained 
next, both approaches can help simplify the conversion 
problem. 

The specialized data management hardware is based on 
the idea of using some kind of an associative memory device, 
a device that can perform a parallel access to the data based 
on its content. Such a device eliminates the necessity for 
organizing the internal structure of a database using indexes, 
hash tables, pointer structures, etc., which are primarily 
used for fast access. As a result, the data can be essentially 
stored in its external logical form, and the data management 
system can use a high level language based on the logical 
data structure only. The conversion process is simplified 
since data is readily available in its logical organization. 
Referring to the terminology used in previous sections, the 
functions of unloading and loading of the database can be 
greatly simplified. Also, no restructuring will be required 
because of a change in database use, since the physical 
database organization can be to a large degree independent 
of its intended use. In addition, the program conversion 


problem is simplified as a result of the program interfacing 
to the DBMS using a high level logical language. 

Similar benefits can be achieved if backend machines are 
used. A backend machine is a special processor dedicated 
to managing storage and database on behalf of a host com¬ 
puter. The primary motive for the backend machine is to 
off-load the data management function from the host to a 
specialized machine that can execute this function at much 
lower cost. From a conversion standpoint, the separation of 
data management functions from the host promotes the need 
for a high level logical interface that provides the advantages 
discussed above. Another advantage is that it is possible to 
migrate from one host machine to another without effecting 
the databases and their management, alleviating the need 
for data conversion if the same backend machine is used 
with the new host. 

Another hardware technology development is mass stor¬ 
age devices, such as the video disks. These devices would 
make it cost effective to store very large databases, in the 
order of 10 10 characters. The problem of converting large 
databases is compounded by cost considerations of proc¬ 
essing this large amount of data. As a result it is likely that 
these databases will tend to stay in the same environment 
for longer periods of time. The use of specialized data man¬ 
agement machines or dedicated backend machines in con¬ 
junction with these mass storage devices can help postpone 
the need for database conversion. 

Finally, we should mention that there is a growing use of 
minicomputers in supporting data management functions. 
DBMSs are now available on many minicomputers, and 
more development is forthcoming. The proliferation of min¬ 
icomputers which support databases will only increase the 
needs for generalized conversion tools. 

Software development trends 

Much of the work over the last years in the data manage¬ 
ment area have concentrated on techniques that clearly sep¬ 
arate the logical structure of the database from its physical 
organization. This concept, called “data independence” was 
introduced to emphasize that users need not be exposed to 
the details of the physical organizations of the database, but 
only to its logical relationships. This led to the development 
of data access and manipulation languages that depend on 
the logical data model only. The effect of this trend is similar 
to that of using specialized data management machines and 
backend machines discussed previously; namely, the sim¬ 
plification of the unload and load functions since the inter¬ 
face to the DBMS is provided at the logical level only, and 
the simplification in program conversion for similar reasons. 

At the user end of the spectrum, it seems reasonable to 
assume that the diversity of data models (network, rela¬ 
tional, hierarchies and other views that may be developed 
in the future) will be required for many more decades. This 
is especially true since there are problem areas that seem to 
map more naturally into a certain model. Furthermore, it is 
often the case that users do not agree on the same model 
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for a given problem area. Obviously this state of affairs only 
accentuates the need to generalize conversion tools that can 
restructure databases from one model to another. Even with 
the development of large scale associative memories, data 
structures will likely provide economic rationales for their 
contrived use. Another possibility is the use of a common 
underlying data model that can accommodate any of the 
user views. However, this approach will still require some 
type of a dynamic conversion process between the common 
view and each of the possible user views. 


Standards development 

There is much work and controversy in developing stand¬ 
ards for DBMS. Standards that are oriented to determine 
the nature of the DBMS are hard to bring about even in a 
highly controlled environment because of previous invest¬ 
ment in application software and database development, and 
because of disagreement. For example, there is still much 
controversy whether the network model proposed by the 
CODASYL committee is a proper one. It seems reasonable 
to assume that there will always be non-standard DBMSS. 
Further, even if such a standard can be adopted, different 
DBMS implementations will still exist, resulting in different 
physical databases for the same logical database. In addi¬ 
tion, one can safely assume that restructuring because of 
application needs will still be necessary, and that changes 
in the standard itself may require conversion. 

A standard that is more likely to be accepted is one that 
effects only the way of interfacing to a DBMS. In particular, 
from a conversion standpoint, a standard interchange data 
form (SIDF) will be most useful. A SDIF is a format not 
unlike a load format for DBMSs. Any advanced DBMS has 
a load utility that requires sequential data stream in a pre¬ 
specified format. If a standard for this format can be agreed 
upon, and if all DBMSs can load and unload from and to 
this format, then the need for reformatting (as described 
earlier) is eliminated. The conversion process can be re¬ 
duced to essentially restructuring only, given that unload 
and load are part of the DBMS function. A preliminary 
proposal for such a standard was developed by the ERDA 
Inter-Working Group on Data Exchange (IWGDE). 32 How¬ 
ever, it is only designed to accommodate hierarchical struc¬ 
tures. Consideration is now given to the extension of the 
standard to accommodate more general structures (i.e., net¬ 
works and rebations). We believe that there are no technical 
barriers to the development of a SDIF, and that putting such 
a standard to use would alleviate a major part of the data 
conversion process. 


Summary 

The reasoning for the points summarized below is given 
in the previous parts of this section. We will only state here 
our assessment of the impact on conversion problems. 


a. Hardware development will increase the need for gen¬ 
eralized conversion tools (in particular, proliferation of 
minicomputers, computer networks, and mass storage 
devices). 

b. The reduction in hardware costs will make conversion 
costs more acceptable. 

c. Special Hardware DBMS machine will simplify the 
conversion process (in particular, for load, unload 
functions, and program conversion) because they pro¬ 
mote interfacing at the logical level. 

d. Software advances will not eliminate the need for con¬ 
version but can simplify the conversion process in a 
similar way as C. 

e. Multiplicity of logical models are likely to exist, thus 
adding to the need of conversion tools between models. 

f. Standards will not eliminate the conversion problem. 
Even if a standard is followed, the implementations 
would be different. Also, it is likely that non-standard 
DBMS will always exist. 

g. Standards can greatly simplify conversion. In particu¬ 
lar, a standard for interchange data form is likely to 
come about (will simplify load and unload and eliminate 
reformatting). 

What can we expect in the next five years and beyond in 
the database conversion area? The state-of-the-art has ad¬ 
vanced enough to date to give hope for generalized tools. 
Within the next five years we can expect more generalized 
conversion systems to become operational, but some addi¬ 
tional work will be required for moving them from one 
environment to another. We can expect to have a standard 
form developed and agreed upon. It will probably take 
longer before manufacturers will see the benefit of adopting 
a standard form and provide load and unload facilities using 
it. However, we can expect them to provide some conver¬ 
sion tools to convert databases from other systems to their 
own. It will probably take as much as ten years before a 
generalized converter is available commercially, with man¬ 
ufacturers adhering to a standard form. Another area of 
concern is the application program conversion that is re¬ 
quired as a result of database conversion. This topic was 
discussed in detail earlier. This is a difficult technical prob¬ 
lem even within one data model, and still requires much 
research. It is hard to expect that a generalized solution for 
this problem will be achieved within the next five years. 
However, this problem will be elevated to a large extent as 
hardware and software development trends promote inter¬ 
facing at the logical level. 
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INTRODUCTION 

The purpose of this paper is to present to the computing 
community an interim report (as of January 1978) on the 
work of the CODASYL Systems Committee in analyzing 
the relationship between database technology and distrib¬ 
uted processing. This paper discusses the following topics: 

• The motivations for incorporating one or more data 
bases into a distributed processing environment. 

• The current objectives of the Committee’s work. 

• The network-oriented and database-oriented compo¬ 
nents, and their relationships. 

• The design choices available to the architect of a dis¬ 
tributed database environment. 

• The technical and administrative issues which arise in 
such environments. 

The Committee hopes that this paper will evoke thoughtful 
comments from its readers. These will greatly aid the Com¬ 
mittee in the preparation of its final report. Comments 
should be directed to: 

Chairman, CODASYL Systems Committee 
Box 1808 

Washington, DC 20013 


PURPOSE/SCOPE OF COMMITTEE’S CURRENT 
WORK 

In accordance with its charter, the CODASYL Systems 
Committee, in January of 1977, undertook a two-year task 


examining the implications of database technology in a dis¬ 
tributed environment. The Committee is focusing its atten¬ 
tion primarily on the impact of extending database manage¬ 
ment techniques to distributed processing environments. 

The scope of this effort is the establishment of a frame¬ 
work for examining these issues, and the formation of base¬ 
line concepts and guidelines which will support the contin¬ 
uing development of distributed database management 
technology. The results of this work will be published at the 
end of this two-year effort as a Systems Committee Tech¬ 
nical Report. 

MOTIVATIONS FOR THE DISTRIBUTED DATABASE 

ENVIRONMENT 

In today’s evolving data processing technology, a number 
of factors have generated a trend from centralization toward 
distribution of data processing functions. Of major impor¬ 
tance to the end user is faster, easier access to the data 
needed for decision making, a need essentially unchanged 
from earlier, more centralized environments. A further con¬ 
cern of the end user is reliability and security. These user 
motivations, coupled with the increasing geographic disper¬ 
sion of the end users within an organization, are generating 
pressures on data processing and corporate management to 
distribute data processing and storage capabilities to the 
location of data origin and/or use. The major benefits derived 
by distributing these functions focus on increased data avail¬ 
ability to the end user and reduced exposure to total system 
failure due to hardware/software failure. 

Historically, the application of computer technology to 
the information processing needs of an organization have 
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tended toward the centralization of data processing and stor¬ 
age facilities. Motivations for such centralization of data 
processing functions have focused on the following: 

• Economies of Scale —the cost of processing and storage 
facilities sufficient to handle the local processing re¬ 
quirements of the end user generally exceeded the com¬ 
bined cost of large, centralized computer facilities and 
a communications facility to provide user access to the 
central facility. 

• Control —security, data integrity, and standards admin¬ 
istration in a centralized environment generally were 
less complex than in a decentralized environment. 

• Availability of Data to the Corporate Level —Although 
not fully satisfying the needs of the distributed end user, 
the centralized environment provided easier generation 
of and access to summary data. 

Today, organizations have grown in size and complexity, 
the users have become more sophisticated in their infor¬ 
mation needs, and the geographic locations of the origin and 
use of data have become increasingly dispersed. As a result, 
the strategy of centralizing the EDP function may run con¬ 
trary to the objective of availability of data to the end user. 
Further, a number of technological factors encourage the 
distribution of data processing and storage facilities. 

These factors include: 

• Lower Cost of Processing and Storage Facilities —re¬ 
cent advances in computer technology and a rapid de¬ 
crease in hardware cost have reduced or eliminated the 
cost advantages experienced through centralization. 

• Cost of Communications —relative to the price/per¬ 
formance gains achieved in computer hardware, com¬ 
munications cost, in terms of price/performance, have 
remained fairly stable, further motivating decentraliza¬ 
tion of data processing and storage facilities in order to 
minimize communications traffic and associated costs. 

• Improvements in Communications Hardware and Soft¬ 
ware Technology —though not as dramatic as witnessed 
in computer hardware and software technology, certain 
developments in communications technology are moti¬ 
vating factors in the trend toward distributing the data 
processing function. For example, the needs for control 
a motivating factor for centralization, can now be ac¬ 
commodated with facilities to “down load” control, 
software, standard application software, or special pro¬ 
cedures to the local processing nodes from a central or 
controlling node. Additionally, communications tech¬ 
nology today readily accommodates the need to access 
one or more nodes in a distributed environment from 
a central or controlling node in order to gather summary 
data. 

In summary, technological and price/performance devel¬ 
opments in recent years are tending toward distribution of 
data processing and storage facilities to the geographic lo¬ 
cation of the origin and/or end use of data. From the per¬ 
spective of end users, distribution of the data processing 
function accommodates the need for accessibility to data 


critical to their decision making responsibilities. This is 
likely to represent a major motivating factor on the part of 
management. Further, a move to a distributed database en¬ 
vironment may realize real hardware/software cost savings 
to organizations. Such savings may represent an additional, 
or perhaps the only, motivating factor. 

The CODASYL Systems Committee, recognizing the 
above factors and concerned with the prevailing perception 
that database technology is applicable and consistent only 
with the philosophy of centralization, has undertaken a 
study of the implications of database technology in a dis¬ 
tributed processing environment. 

WHAT IS A DISTRIBUTED DATABASE 

ENVIRONMENT 

Considerable attention has been given recently to com¬ 
puter networks and to database technology, but little to the 
application of database technology in a network environ¬ 
ment. The traditional single-site orientation of DBMS and 
databases must be extended to operate in the context of a 
network of computer facilities. To provide a basis for dis¬ 
cussion and investigation, the committee first laid a careful 
foundation of terminology. The following paragraphs define 
network and database related aspects of a distributed proc¬ 
essing environment. 

The network provides the underlying configuration of 
computer systems and communication facilities within 
which data is stored, DBMS’s operate, and users access 
data. A node in the network consists of computer processing 
facilities (ranging from a large multiprocessor computer to 
an intelligent terminal) and an associated operating system 
sufficient for executing user and DBMS processes (pro¬ 
grams, queries, etc.). In addition, data and its definition may 
be stored at a node. The precise structure of a node is an 
architectural design choice independent of the manner in 
which the node is connected to other nodes and the extent 
of geographical separation. 

A communications facility is the collection of processes 
and physical facilities which interconnect the nodes. The 
communications facility includes knowledge of the physical 
location of each node, the physical path connections be¬ 
tween the nodes, and the protocols to be used in sending 
messages between nodes. Processes in the communications 
facility will accept a message from one node and deliver it 
to another node or broadcast it to some or all other nodes. 
Two nodes may be connected directly or indirectly through 
other nodes. A network access process (NAP) exists at 
every node as the interface between processes at the node 
and the communications facility. The NAP is that portion of 
the communications facility which executes on the process¬ 
ing facilities of a node. 

A traditional single-site database environment (See Figure 
I) 1,2 consists of a database, a DBMS, a database definition 
(“schema”), and a user schema.* Placing these data-related 


* User schema broadly refers to the user’s view of the database. CODASYL 
has called this a “subschema”, but, in general, the only requirement is that 
a mapping exist between the user schema and the “schema.” It is not strictly 
a subset of the schema. 
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Process 

Data, System Information (userschema need not physically exist) 
direct transfer of data/language statements/control 

additional transfers (of less interest to us) 
relationship 


Figure 1—Conventional single-site database environment 
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components at the nodes in a network environment produces 
a distributed database environment. The primary architec¬ 
tural design choices relate to the distribution of databases, 
and the distribution of DBMS functions. All the databases 
may be stored at one central location (the traditional single 
site), they may be stored at different nodes, a given database 
may be broken down into pieces (“partitioned”) with the 
pieces stored at different nodes, or finally, multiple copies 
of a database or its pieces may be stored at multiple nodes 
(“replicated”). Whenever a database (or part thereof) is 
stored, there also must exist appropriate data definition and 
DBMS capabilities to access and process the data. When 
the user request and the target database exist at different 
nodes, DBMS functions can also be distributed and per¬ 
formed at multiple nodes. 

The integration of database technology and a network 
environment leads to new problems and the need for new 
functions. Perhaps the most important need is for some 
network-wide intelligence concerning the node location of 
all the databases in the system, their partitions and their 
replications. The network data directory serves this func¬ 
tion. 

When the system receives a user request to access data, 
a new DBMS function must first determine what data must 
be accessed and the node or nodes on which it resides. Then 
it must communicate with another DBMS. Increased het¬ 
erogeneity within the distributed database environment, i.e., 
multiple different DBMS’s, necessitate a translation of the 
request. Data conversion may also be necessary if the data 
is stored in different forms, according to different data struc¬ 
ture models or on different hardware/storage devices. These 
are some of the key technical problems to be addressed by 
the data processing community in the coming years. 

DESIGN ALTERNATIVES 

Three levels of perspective 

The Committee recognizes three distinct, but nevertheless 
interdependent, perspectives in providing design altema- 
. fives for a distributed database environment. 

• Each component of the environment and the interrela¬ 
tionship of components. (See Figure 2) 

• The study, in static state, of a node in the distributed 
environment including its various alternatives for con¬ 
figuration. (See Figure 3) 

• The specification of user interaction in a distributed 
environment, emphasizing the sequence of events and 
the execution of functions, creating a dynamic model 
of the environment. 

Components and interrelationships 

There are five major components within a node of a dis¬ 
tributed database environment as illustrated in Figure 2. 
Non-distributed databases have three of these—the database, 


the DBMS, and the user process. A distributed database 
environment requires two additional components—a net¬ 
work DBMS and a NAP. Figure 2 shows three types of 
nodes: 

• A “complete” node with all five basic components. 

• A “user” node with only a user process accessing data 
in the network. 

• A “data” node with only those components necessary 
for data to be made available throughout the network. 

Figure 3 provides a more detailed picture of “complete” 
node. To convert a conventional database environment (See 
Figure 1) to a distributed database environment requires the 
addition of the following components: 

• Network DBMS 

• Network Data Directory 

• NAP (described above) 

With a centralized database, which is the case in a single¬ 
site data processing environment, all the intelligence relating 
to the database is in one place. A conventional data defini¬ 
tion can provide all the information needed for a DBMS to 
locate and process the stored data. 

As soon as data becomes distributed—single, whole da¬ 
tabases at different nodes, partitioned databases, or repli¬ 
cated databases—there must exist some information within 
the system which indicates where the different databases, 
their partitions, and their replications are stored. This is the 
primary role of the network data directory. 

Network data directory 

The network data directory contains information indicat¬ 
ing the nodes at which the various units of data reside within 
the distributed processing environment. For example, given 
a file name, the network data directory provides the node 
or an indirect reference to the node(s) at which the file 
resides. It does not contain information about the physical 
location of nodes or the routing between nodes; this infor¬ 
mation is part of the communications facility. 

Network database management system 

If we assume that a DBMS only includes those functions 
which relate to a local database and that it has no cognizance 
of any other data nodes (i.e., a conventional, single-site 
DBMS), then all other new functions needed in a distributed 
environment can be packaged in a new system called the 
Network Database Management System (NDBMS). In the 
development of a distributed processing environment, it may 
be desirable to integrate the NDBMS functions with the 
DBMS, with the NAP, or leave as a separate package. We 
will be more concerned with the nature of the function and 
its information inputs, than with any particular packaging 
approach. 





Figure 2—Data and users in a distributed database environment 
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Data, System Information (userschema need not physically exist) 
direct transfer of data/language statements/control 
additional transfers (of less interest to us) 
relationship 


Figure 3—A “complete” node in a distributed database environment 
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The network DBMS includes at least the following func¬ 
tions: 

• Intercept a user request and determine where to send 
it for processing, or what nodes must be accessed to 
satisfy the request. 

• Access the network directory or at least know how to 
request and use the information in it. 

• Coordinate the processing and response to a user re¬ 
quest if it spans nodes, that is, the target data exists at 
multiple nodes. 

• Function as the communication interface between the 
user process, the local DBMS, and DBMS’s at other 
nodes (via the NAP). 

• Provide data and process translation support in a het¬ 
erogeneous distributed database environment. 


Design choices and distribution alternatives 

The specification of design choices and distribution alter¬ 
natives for various distributed database architectures re¬ 
quires a model that will facilitate the study of database 
related distribution alternatives. If a major component does 
not exist as a single copy at a single node, then there are 
multiple alternative design choices as to its distribution to 
nodes. 

• Replication Versus Partitioning of Components —the 
replications of a distributed component (or any piece of 
a component) are functionally identical copies of the 
component. Replication introduces the issue of update 
synchronization. 

The partitions of a distributed component are differ¬ 
ent and non-overlapping. Under partitioning, the whole 
of a component consists of multiple pieces which the 
system must be able to associate. The system must 
know the relationships among the pieces and be able to 
manage the whole component and its pieces. 

• Homogeneous Versus Heterogeneous Implementa¬ 
tion —a central issue relative to the replicated and par¬ 
titioned distribution alternatives regards the relative de¬ 
gree of homogeneity/heterogeneity of the supportive 
components (namely the DBMS, schema, non-schema 
and system information) across nodes. If the corre¬ 
sponding supportive components at each node are dif¬ 
ferent (heterogeneous) then the complexity and tech¬ 
nological difficulty implementing the system increases. 
If all components are the same (homogeneous) then it 
is easier to implement and operate the environment. 
Given today’s technology, the ability to design, imple¬ 
ment and operate a distributed environment is greatly 
facilitated by “standardizing” components at each 
node. Thus, for example, utilizing the same DBMS uni¬ 
formly at every node greatly simplifies the design of a 
distributed environment. 


Figure 4 summarizes these design choices: 

COMPONENT 



Replicated 

Partitioned 
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The component exists in 
multiple identical imple¬ 
mentations 

The component exists 
only once in the system 
and has identical imple¬ 
mentations 

w 

Cm 

S Heterogeneous 

The component exists in 
multiple copies, but in dif¬ 
ferent forms of implemen¬ 
tation 

The component exists 
only once in the system, 
but varies in its implemen¬ 
tations 


Figure 4—Alternatives for distribution of a component 


These simple rules are complicated by observations of 
existing distributed environments. 

• Hybrid distribution alternatives are possible as are time 
variant ones. 

• Not all distribution alternatives are relevant to realistic 
systems. 

• Not all combinations of component distributions are 
internally consistent or useful. 

• Complex interdependencies exist. 


TECHNOLOGICAL AND ADMINISTRATIVE ISSUES 

Benefits from the distribution of data and its management 
are not without cost. Benefits resulting from the dispersal 
of data processing and DBMS capabilities must be balanced 
against the new technological and administrative problems 
which result in costs. The solution of these problems will 
demand a significant cost commitment in terms of additional 
hardware, software and communications facilities. Simi¬ 
larly, distribution will impact at the administrative level. 
Data control is complicated since the distributed environ¬ 
ment and administrative control must be more highly coor¬ 
dinated. For the purposes of its report, the Committee dis¬ 
tinguishes between technological and administrative 
implications. 

The Committee has thus far in its deliberations identified 
a number of issues which sharply focus the technological 
implications of distribution. Each of these is briefly de¬ 
scribed. 

Initial implementation problem 

The first step in planning for a distributed environment 
involves careful engineering of the transition from existing 
operations to the new. Managing the conflicts of heteroge¬ 
neity creates a significant problem. Often disparate database 
management systems may be in use, different processing 
hardware may be installed, and user languages may be in¬ 
compatible. Conversion from the existing environment to a 
distributed environment is a technological issue. 
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Access control 

A centralized DBMS facilitates the implementation of se¬ 
curity restrictions. Through security mechanisms, protec¬ 
tion is afforded against unauthorized, deliberate or inad¬ 
vertent access to data in the system. One important 
mechanism used to achieve this goal is access control. In a 
distributed environment, access control is more difficult to 
implement because of the necessity of deciding where to 
locate the control mechanism. The vulnerability of dispersed 
data may be a real issue. In the extreme, it may be necessary 
to locate access control with each distributed component, 
a costly decision. On the other hand, centralizing access 
control may unduly burden the system with performance 
inefficiencies. However, if highly sensitive data is localized 
to specific nodes, higher assurances of strongest access con¬ 
trol can be made. 


Concurrency 

A DBMS must include provisions for the control of con¬ 
current multiple updates to a single record occurrence. In 
a distributed environment, the record occurrence itself may 
be partitioned across nodes. The control of concurrency can 
then become a more complex issue. 

Synchronization 

A DBMS takes into consideration the management of 
known redundancies such that a single update affecting mul¬ 
tiple copies of the same record occurrence are effected in a 
manner transparent to the user and such that the database 
is always at the required level of consistency. The geo¬ 
graphic dispersion of replicated copies makes this task more 
difficult. 


Translation problems 

In a distributed environment the number of conflicting 
logical views increases with the amount of heterogeneity in 
the network. The ability to effect the translation of data, 
processes, programs, user requests, and procedures from 
one node to another is a basic requirement of a distributed 
environment. The ability to implement transparent transla¬ 
tion mechanisms presents a technological challenge. 


Continuity of operation 

An integral part of a DBMS’s objective is the ability to 
operate continuously. This entails specific provisions for the 
recovery of data and the restart of operations. In a distrib¬ 
uted environment this task becomes more difficult when 
applied to partitioned distribution, and when considered in 
conjunction with the synchronization issue. 


Performance issues 

The ultimate goal is to provide information services to the 
user. Success in measured, in part, in terms of delivery of 
timely results. The underlying rationale of distribution is 
availability of data. If the implemented system cannot per¬ 
form efficiently enough to meet response requirements, the 
service goals are not met and the system fails. When distri¬ 
bution of components is achieved, there is a risk that the 
communications overhead incurred in the interest of system 
integration and control will degrade performance sufficiently 
to cause the user to question the responsiveness of the 
system as a whole. The balance between distribution and 
control will thus be a key performance issue. 

Administrative issues 

The design, implementation and operation of a distributed 
environment will be impacted by a number of administrative 
issues. The fact that these issues cannot be resolved at a 
technical level make them all the more important and de¬ 
serving of attention. 

Constraints on data flow 

The distributed environment introduces the potential for 
increased amounts of data storage and flow at and across 
geographically dispersed locations. The assumption that 
data can be moved subject only to economic considerations 
is no longer valid. Constraints on the storage and flow of 
data may be expected from a number of sources, such as: 

• Legal —governments will legislate constraints on data 
flow in the form of tariffs and regulatory law. 

• Privacy —privacy requirements and restrictions will 
have an effect on the storage and flow of data. 

• International Law —the flow of data across interna¬ 
tional boundaries is just beginning to be examined. Re¬ 
sponsibility for control, authorization, nature of trans¬ 
mission, rights to impose legal restrictions, traffic 
constraints, etc., remain to be established. 

Coordination and control of data 

The administration of data resources is recognized as an 
important prerequisite for successful implementation of 
shared use of data. Distribution of data complicates this 
issue. The development and enforcement of standard prac¬ 
tices is more difficult with dispersion, the coordination of 
common data descriptions is a problem when users are geo¬ 
graphically dispersed, and the resolution of conflicting data 
needs becomes more tedious and time-consuming with dis¬ 
tribution. 

Organizational issues in administration of data 

The emergence of an organizational entity (Data Admin¬ 
istration) to address the foregoing issues has provided users 
with an administrative mechanism to deal with the problem. 
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In a centralized database implementation, the data admin¬ 
istration function has often been organized to be controlled 
by a senior level data processing officer. In a distributed 
environment, however, data administration may be poten¬ 
tially exercised at each node in the network. Issues of or¬ 
ganizational placement, interaction and coordination must 
be carefully scrutinized. 

User acceptance 

The distributed environment must, in the final analysis, 
serve the user. There are a number of phenomena which 
may cause concern in this regard. 

• Resistance to change. 

• Conversion from existing user procedures. 

• Potential risk of complexity. 

• Organizational realignment. 

In a distributed environment each user at every node must 
find it attractive to participate or the system will fail. 

CONCLUSION 

The impact of “distributed processing” is already mani¬ 
fested in many sectors of the EDP community. The impact 
on database technology is significant; the subject of current 
deliberations of the CODASYL Systems Committee is forg¬ 
ing a framework for dealing with these new architectural 
concepts and is establishing a basis for dealing with the topic 
on common ground and with clear understanding of the 
issues and considerations. 
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Commentary on 

the CODASYL systems committee’s interim 
report on distributed 
database technology 

by CHARLES BACHMAN 

Honeywell Information Systems 
Waltham, Massachusetts 


The author has reviewed the Interim Report on Distributed 
Database Technology and is responding from the vantage 
point of the ANSI/X3/SPARC/Study Group on Distributed 
Systems. This study group is attempting to establish an 
architectural approach which will put all aspects of distrib¬ 
uted systems into perspective, including distributed data¬ 
bases. The author has been studying this problem as part of 
Honeywell Information System’s investigation for the last 
three years. 

My first comments must be addressed to the overall doc¬ 
ument which is well written and conveys its message well. 
I am in agreement with everything in it and will address 
myself to those aspects which it does not cover at the pres¬ 
ent time. As an interim report, one cannot criticize it se¬ 
verely for having some gaps. It has been circulated for 
comment and is soliciting comments so that the final report 
may be more accurate and more complete. I recommend the 
reading of this report to all interested parties and urge each 
reader to comment so that the final report will be as com¬ 
plete as time permits. 

The interim report lists a number of motivations for dis¬ 
tributed systems. All are valid in one situation or another. 
There are a couple of additional reasons which I would like 
to add to the list. The first reason is a human factor reason. 
This is to permit some element of the business to physically 
possess the part of the corporate database which holds its 
part of the data. This gives them the responsibility and the 
satisfaction of updating their database without removing it 
logically from the required access by others. Another human 
factor reason relates to the distribution of the database to 
reduce its vulnerability to strike action or acts of terrorism. 
Another reason is the matter of database size vs. computer 
power. There is no computer on earth which can by itself 
handle all the data processing for a large corporation. There¬ 
fore, processing must be distributed and is already distrib¬ 
uted for most of the FORTUNE 500 companies. In most 
cases they are already physically distributed but not logically 
integrated and will not be logically integratable until “dis¬ 
tributed database facilities’’ become available. 


There should be one item added to the list of reasons for 
the centralization of data processing. This is to achieve a 
concentration of expertise sufficient to run a large data proc¬ 
essing operation. From an administrative point of view, this 
is very important. Learning how to distribute the expertise 
along with distributing the processing and database may be 
a real roadblock. 

From the vantage point of the ANSI Study Group on 
Distributed Systems, the report seems to lack a certain ar¬ 
chitectural overview which would put distributed database 
management into perspective. The ANSI distributed sys¬ 
tems reference model begins with the notion that the infor¬ 
mation processing aspects begins with the notion that the 
information processing aspects of an enterprise should be 
viewed as a network of cooperating workstations. Each such 
workstation is a “location” where a certain kind of work 
takes place. This workstation may be manual, industrial or 
computerized. This workstation supports one or more pro¬ 
cesses which execute the procedures of the workstation and 
which primarily access the locally stored data. Access to 
remoted stored data is acceptable, but known to be slower 
and more expensive. The processes of the workstations 
cooperate by exchanging messages which request certain 
actions and transfer such data as appropriate. 

Where the process and the data it accesses are in the same 
machine, data access operates essentially as it did before it 
was distributed. Where the process and the data are in 
separate machines, more elaborate mechanisms are neces¬ 
sary. Those described in the CODASYL interim report are 
representative. One thing which appears missing from the 
report is any discussion of how processes and data would 
be partitioned and set up in the various machines t in the 
network. While this is a packaging issue, my current guess 
is that it is a critical one. Probably, as critical to distributed 
database access as the “clustering” concept is to the per¬ 
formance of our current centralized database. Simply speak¬ 
ing, clustering is the placing together in the same page as 
many records of a set occurrence as possible. Where one 
set has been chosen in favor over another because of the 
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anticipated frequency of access. Frequency of access is also 
the heart of the database vs. process distribution questions. 
It has been said that there are two approaches: (1) take the 
data to the process and (2) take the process to the data. The 
carefully preplanned distribution of processes and the data 
which they are most likely to access to the same machine 
is the most important step which avoids much of the proc¬ 
essing burden associated with distributed processing. 

Within the ANSI reference model, three alternate sche¬ 
mas have been discussed to achieve access to distributed 
databases. From our point of view, the picture presented by 
the CODASYL interim report describes the most difficult to 
design and implement. It will also be the least efficient under 
certain operating circumstances. The three approaches dis¬ 
cussed are characterized as; 

(1) cooperating user workstations 

(2) cooperating storage management workstations 

(3) cooperating database management workstations 
(CODASYL) 

All of these approaches assume that a facility for process to 
process exchange of information is a basic capability of a 
distributed system. The difference in the three approaches 
is who is talking to whom and what is the nature of the 
information exchanged. 

The cooperating user workstation approach assumes that 
the users are knowledgeable of the data file vs. process 
(workstation) configuration. Thus, if a process cannot get 
what it needs from the local database, it will formulate a 
message to be sent to another workstation which is co-lo- 
cated with the data which needs to be processed. When the 
message arrives at the workstation, a process of that work¬ 
station will access the message and carry out whatever local 
access and updating is necessary and respond with an an¬ 
swering message. When workstations are considered as ele¬ 
ments over which the total work load has been divided, then 
it does not make much difference if one is designed to 
approve a credit request for a new sales order or whether 
one is designed to answer questions about the Minneapolis 
inventory. It is only a slightly further move to design a 
workstation which receives data manipulation commands as 
a message text, executes these commands in terms of its 
local database management system, and sends a response in 
accordance with its success. 

The cooperating storage management workstation ap¬ 
proach is based upon the fact that most of our database 
systems are built as virtual memory systems and do “page 
turning” independent of the operating system and hardware. 
When a page is requested by the database management 
subsystem the storage management subsystem asks whether 
the page is in a main memory buffer or whether it must be 
fetched from the disc? If it must come from the disc, then 
the question is which disc drive and address? With a dis¬ 
tributed storage management system, the third question 
must be asked, my disc or one managed by some other 
storage management subsystem? If it belongs to someone 
else, then a message must be formulated with the request. 


A response message will either return the page, state the 
process must be waited until the page is available, or stating 
that a deadlock would result if the process were to be waited. 
The cooperating database workstation is essentially that one 
described in the CODASYL Interim Report. 

The CODASYL report discusses several kinds of control 
tables: network directory, concurrency control, integrity 
control and security control. There is a discussion appearing 
several places in the report which questions whether these 
tables should be centralized or distributed. From my point 
of view, there is only one answer: distributed. The central¬ 
ized approach has all the problems which the distributed 
database system is attempting to side-step, e.g., the vulner¬ 
ability of the entire system to the failure of one element, the 
congestion problem, etc.). What must be done is to give 
each node the information about its local resources, process 
and only information about remote objects of local interest. 

One note about the concurrency control is relevant, the 
problem in a distributed system takes on complexity beyond 
just the aspect of multi-nodes. Most concurrency control 
systems have a secondary responsibility to be sure that two 
or more processes are not waiting for each other in order to 
access data (deadlock). With the introduction of message 
exchange between processes, it is possible for process (1) 
to be waiting for a message from process (2) which is waiting 
for a record held by process (1) . . . deadlock! It also must 
be considered that a process may be waiting upon a terminal 
operator action. If that terminal operator could possibly be 
waiting on the action of another process in the system, then 
that anomie must also be included in the path of waiting 
processes to determine if deadlock exists. 

The question was raised concerning the effect upon con¬ 
currency if a record were to be split between nodes. This 
isn’t a problem because the assignment and wait logic can 
work at any level of granularity. It could work at the level 
of the logical elements: item, item group, segment, record 
or file. It could work at the level of physical elements: page, 
group of pages, track, cylinder, or extent. As long as the 
control is maintained non-ambiguously, it will work. The 
only difference is the smaller element requires more book¬ 
keeping and the larger elements force processes to wait and 
possibly be recovered from deadlock when the effective 
access desired would have caused no interference. 

The text described two basically different forms of distri¬ 
bution: distribution of parts of the only copy and distribution 
of copies of the whole database. Where updating is a perti¬ 
nent issue, the distribution of copies seems less efficient 
than distributing parts of the only copy and getting it updated 
at its only location. We have seen several large systems in 
the field which have chosen a variation which is worth 
describing. It could be considered as a form of the multiple 
copy approach. In this case, major portions of the file are 
copied and partitioned to be sent to a number of transaction 
processing machines. These transaction machines handle 
the on-line daytime processing of their partition of the da¬ 
tabase. All on-line transactions are also sent to a centralized 
system where they are batched with additional off-line trans¬ 
actions and reprocessed in a nightly batch run. At the end 
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of the nightly run, the updated centralized database is again 
copied and updated partitions are sent to each transaction 
machine. 

So much for the extension of the generalities. The real 
question is where do we go from here. I see the only satisfac¬ 
tory long term answer in the form of a set of international 
standards which permit all kinds of computers, software, 
communication facilities and terminals to be joined together 
to form an extended network of cooperating workstations. 
I say extended because the workstations served will need to 
communicate within companies and between companies and 
across international borders. No one vendor or one country 
is large enough, smart enough, or influential enough to go 
it alone. This is the reason I have supported the ANSI effort 


and the ISO effort to move toward this standard. We have 
set ourselves an impossible schedule to reach the goal of 
international standards. The end of 1978 is the goal for an 
internationally accepted reference module (levels, protocols 
and interfaces). The end of 1980 is the goal for the first new 
protocol and interface standards which will support the 
interprocess exchange of information. Working groups are 
being established to study and prepare specifications. One 
of these is concerned with the distributed database access 
and control. I expect that the CODASYL System Committee 
with its members and report will make a significant contri¬ 
bution to making those standards. Are you ready to move 
and move rapidly? Are you ready to help write and negotiate 
these standards? 
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Computer architecture 


The recent advances in Computer Technology, providing unlimited possibilities 
for an ever expanding growth of the application spectrum, have created substan¬ 
tial research activities. The technical sessions in the architecture area of this 
conference are conceived and planned to expose the profound advances in hard¬ 
ware technology and also to reveal their significant impact on the current and 
future systems architecture. This series of sessions presents the state of the art 
and also some of the thought provoking challenges to the designers of the future 
systems. 

The session the “Impact of Semiconductor Technology on Computer Archi¬ 
tecture” traces its impact of semiconductor technology on the computer archi¬ 
tecture and the emergence of microprocessor systems. It also highlights a dis¬ 
cussion on how the future data processing technology would be influenced by 
the semiconductor technology and the associated problems. The evolution of this 
technology and future projections are discussed by the leaders of this emerging 
technology. 

The sessions on “Evolution of Architecture” trace the evolution of architec¬ 
tures of well known computer families. These sessions articulate the major chal¬ 
lenges and accomplishments of these systems. They also discuss the lessons that 
these systems have taught and how the future systems would be built. These two 
sessions would define certain cautious guidelines for the future systems. 

In the future the user is expected to be his own architect. By specifying his 
requirements the future user would be able to configure his system from the off 
the shelf subsystems. In order to do this certain tools and techniques are needed 
by the user. The session on “User Impact on Architectures” explores such issues 
related to user involvement. 

In order to define the directions for future large scale architectures it is essential 
to look back and assess the effectiveness of existing large scale systems. Session 
on “Large Scale Computer Architectures” looks at the evolution of large scale 
systems and discusses how effectively these systems have satisfied the user 
needs. 

The dramatic reduction in hardware costs due to the LSI technology make it 
economically feasible to expand the application spectrum and to cover the areas 
which are hither to unexplored by developing special purpose architectures. 
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Session on “Special Purpose Computer Architectures” highlights architectures 
for one of the important application areas—pattern recognition. 

In spite of tremendous advances in computer technology the existing intracom¬ 
puter standards are in chaos and some are confusing. With the result there exists 
a large degree of incompatability between the subsystems of various manufac¬ 
turers thus imposing several artificial constraints on the designers who use these 
subsystems. The session on intra-computer standards will extend a discussion on 
the impact of standards on computer architecture. Organizations involved in 
developing these standards will discuss their current thoughts and focus on how 
uniform standards could be evolved that would benefit both the user as well as 
the computer architect. 



Intra-computer standards 


by TSE-YUN FENG 

Wayne State University 
Detroit, Michigan 


The need of standards on many industrial products is quite 
easy to understand. Ideally, we would like to have various 
parts of a product interchangeable. From a user’s point of 
view standards provide him with desirable convenience and 
reductions in product cost as well as costs for maintenance 
and repair. In some respect, even manufacturers stand to 
gain from standardization. Yet, the progress in standardiza¬ 
tion on various products has been slow. This is particularly 
true ih the case of computer standards. 

Since the introduction of the first electronic digital com¬ 
puter some thirty years ago the standards activities on com¬ 
puters have been relatively small. There are several factors 
that may contribute to this phenomenon. One is that until 
recently computer systems have been quite bulky involving 
many components and circuits that are proprietary or pat¬ 
ented. The number of major companies manufacturing com¬ 
puters are relatively few and each of them has its own 
system of standard equipment. Also, the maturity of a prod¬ 
uct is usually a prerequisite for standardization, but the 
rapid advances in computer technology prevent the stand¬ 
ardization in many areas. It is thus not surprising to find 
that much of the recent national standards effort has been 
mainly in the language area since they are relatively machine 
independent. Languages such as FORTRAN, APT, 
COBOL, PL/1, BASIC have been standardized by the 
American National Standards Institute (ANSI). In addition, 
there are also activities on character sets, and information 
processing vocabulary. Recently, IEEE Computer Society 
took the initiative in proposing and developing software 
standards. These projects are on Software Development, 
Software Quality Assurance, and Software Documentation. 


On the hardware side, the standards activities have also 
been limited to those that are not closely related to the 
central components of computers. They are mostly on mag¬ 
netic tapes, disc packs/cartridges, codes, character recog¬ 
nitions, and interface circuits. Recent advances in integrated 
circuits make it possible to have a large number of memory 
bits and their associated circuits implemented on one chip. 
A number of these chips can then be interconnected to form 
a larger memory. Furthermore, such large scale integration 
also permits designs for a small processor to be implemented 
on a chip. The microprocessor chips together with memory 
modules can be used to form larger computer systems. The 
impact of the integrated-circuit chips on future computer 
architecture is quite evident. Several standards projects on 
semiconductor memories and microprocessors have also 
been initiated by IEEE Computer Society with the partici¬ 
pation of both manufacturers and users. On semiconductor 
memories, the projects involve Electrical Description of 
Semiconductor Memory Devices, Test Pattern for Testing 
Semiconductor Memories, and Thermal Properties of Mem¬ 
ories and Other IC’s. In addition, a number of new standards 
projects relating to semiconductor memories are being ini¬ 
tiated. In the microprocessor area three new standards pro¬ 
jects, Floating Point Format and Algorithms, Assembly Lan¬ 
guage Syntax and Lexicography, and Relocatable Object 
Module Formats are in progress. 

It should be noted that there are also parallel efforts on 
computer standards by the federal government, the military, 
as well as Joint Electron Device Engineering Council 
(JEDEC), the International Purdue Workshop, etc. 




Software standards—With hints of their relation to 
computer architecture 
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INTRODUCTION 

Software standards can help the developer, the user, and 
regulatory agencies in facilitating communications, in ex¬ 
tending software life, in broadening the applicability of pro¬ 
grams, and in reducing the personnel and organizational 
reorientation required in transitioning from one project to 
another. 

Communications are facilitated by use of standardized 
terminology, by use of standards for development method¬ 
ology (e.g., structured programming), and by standardized 
test reporting to name just a few pertinent areas. Facilitating 
communications implies not only that misunderstandings are 
avoided but also that the software procurement cycle can 
be shortened and procurement costs reduced because both 
contracting parties may from prior experience know their 
obligations when a standard is invoked. Use of a standard 
also reduces the cost of software development because the 
allowance for coping with unfamiliar specifications can be 
reduced. 

Software standards can extend the useful life of programs 
by facilitating adaptation to new versions of operating sys¬ 
tems, by making it possible to transfer the program to a new 
computer, and by making maintenance and updating of the 
program itself a lot easier and cheaper. The broadening of 
a program’s applicability is made possible by the features 
cited above plus uniformity among users of program format, 
test, and test reporting requirements that can be accom¬ 
plished by standards. 

Much time is required during the start-up phase of critical 
programs in order to train otherwise experienced analysts 
and programmers in specific contractually imposed design, 
coding, or documentation requirements. Allowances have to 
be made for errors in complying with unfamiliar require¬ 
ments, and at times management may be reluctant to bid on 
a software development that is within its normal range of 
business interests simply because of uncertainty regarding 
a test or documentation requirement that had not previously 
been encountered. Software standards could reduce this 
wasted effort in transitioning from one project to another. 

In view of these benefits one must ask why the few ex¬ 
isting standards are not more widely employed and why 
there is not more effort at implementing software standards. 


Part of the answer lies in the newness of software engineer¬ 
ing as distinct from programming for one’s own use. The 
latter implies the translation of a defined sequence of oper¬ 
ations into machine-readable form, an activity that was best 
managed in a local context and to which standards are about 
as applicable as they are to the writing of business letters. 
Software engineering recognizes that requirements analysis, 
test, validation, program maintenance and related tasks, in 
addition to programming proper, are necessary to provide 
the computing service that the user expects and needs. This 
broader activity involves the interaction of many groups, 
extends over a long period of time, and is in scope quite like 
a construction project. Hence, standards are appropriate to 
guide the activities. 

But together with the recognition of the benefit of stand¬ 
ards must also come the recognition that they involve costs. 
A standard for documentation may require a more detailed 
format than is essential for a given application, and a soft¬ 
ware quality assurance standard may impose access control 
that will be perceived as burdensome by personnel that ar¬ 
ranged the mounting of their own test tapes as they saw fit. 
In the context of computer architecture it is appropriate to 
point out that just about every software standardization ac¬ 
tivity will increase memory and throughput requirements. 
For example, standards for test or test documentation may 
invoke analysis tools that slow down the execution of the 
primary program. So right at the outset let it be said that 
whatever can be done to increase the speed of computers 
and the size of the easily accessible memory will be welcome 
for the application of software standards. 

In terms of scope, software standards can be divided into 
three broad categories: language standards, software engi¬ 
neering standards, and software implementation standards. 
Each of these classes will be separately discussed below. 
There is another important classification of standards by 
issuing agency: voluntary industry standards, semi-official 
standards (e.g., those promulgated by international agen¬ 
cies), federal standards, and military standards. This paper 
will be principally concerned with non-governmental stand¬ 
ards. 

Finally, the common Use of the term standards encom¬ 
passes some documents that are actually recommended 
practices, guidelines, or specifications. These documents 
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obviously have a lesser enforceability or more restricted 
applicability than true standards, but, in view of the rather 
limited effect of any software standards on computer archi¬ 
tecture, these distinctions are not too important in the pres¬ 
ent context. 


LANGUAGE STANDARDS 

Standards promulgated by professional organizations are 
presently applicable only to High-Order Languages 
(HOLs). 1-8 As such, their effect on computer architecture 
is indirect; clever compiler designers, if necessary, can cope 
with almost any machine instruction set. Military agencies 
may be motivated to standardize the assembly level instruc¬ 
tions for certain classes of computers, primarily those that 
process tactical software. Implementation of such standards 
would have a much more direct effect on architecture, but 
discussion of this event is beyond the scope of this paper 
which emphasizes voluntary compliance. 

We should also be careful to point out that the effect HOLs 
have on computer architecture is to a large measure inde¬ 
pendent of the existence of language standards. Thus, the 
adoption of features such as general register organization and 
floating-point processors that are a boon to processing FOR¬ 
TRAN object code preceded the adoption of the ANSI FOR¬ 
TRAN standards. Standardization of HOLs may neverthe¬ 
less be significant in this connection because it increases the 
portability of code and makes the benefit of architectural 
features that support HOLs more visible to the user. Since 
source code written in a standardized HOL can be executed 
on any computer for which a compiler written to that stand¬ 
ard exists, it can become quite apparent to a user interested 
in such matters how both compile time and run time are 
affected by the computer organization. Hence, standardiza¬ 
tion of HOLs in general is a positive factor in promoting 
architectural features that support processing of HOL-de- 
rived code. 

So far the discussion has dwelled on use of compilers for 
translating a program coded in a standard HOL for gener¬ 
ating the instructions that ultimately are operated on by the 
computer. Another approach for handling HOL source code 
is through direct-execution machines. Considerable research 
in this field has been under way for some time but so far 
leading to full-scale development of only a few direct-exe¬ 
cution computers. 9 Among many factors responsible for the 
slowness of this development may have been the prolifera¬ 
tion of language “dialects”, requiring translation or other 
special processing before existing source code could be 
processed on a direct-execution machine. HOL standardiza¬ 
tion may remove at least this factor among the obstacles to 
acceptance of architectures for direct execution. 

Block-oriented languages, such as ALGOL, benefit par¬ 
ticularly from provision of stack features in the computer. 
Further standardization of block-oriented languages may 
therefore become a factor in promoting stack architectures. 10 

The general tendency toward increased use of standard¬ 
ized HOLs (likely to be heightened by a recent DoD direc¬ 
tive) will tend to benefit those computers that can efficiently 


process code derived from them. However, it must also be 
realized that going from assembly code to HOLs will require 
more memory and greater processing speed if the same 
computational capabilities are to be provided. Thus, the 
previously mentioned effect that standardization demands 
faster computers with larger memories is evident in the area 
of language standards. 

SOFTWARE ENGINEERING STANDARDS 

In the broader sense, dictionaries for information proc¬ 
essing that have been available for some time 11 could be 
considered software engineering standards. Also, some De¬ 
partment of Defense acquisition management documents 
contain material related to software engineering standards. 12 
But specific software engineering standards are only in the 
draft stage, e.g., a Military Standard for Tactical Software 
Development was circulated as a draft in June 1977. It is also 
interesting to note that the Federal Information Processing 
Index provides for software engineering subjects, e.g., op¬ 
erating procedures and operating systems, but that there are 
at present no entries under these headings. 

Effectiveness of software procurement and development 
suffers, and in some cases the introduction of computers for 
critical tasks is being hampered, by this lack of standards. 
The IEEE Computer Society Subcommittee on Software 
Engineering Standards is now active in preparing three 
standards in this area: software development, software qual¬ 
ity assurance, and software test documentation. A special¬ 
ized terminology for software engineering has also been 
drafted by this committee. 

The software development standard is expected to require 
a modular organization, some form of structure program¬ 
ming, and “readable code.” These requirements do not eas¬ 
ily translate into effects on machine architecture but they will 
in general result in an expansion of code aver that generated 
without them and hence again will demand greater memory 
capacity and speed. 

Standards to be drafted for the quality assurance area may, 
among other things, be concerned with controlling access to 
software and with recording of all executions of a program 
during test or certain other critical phases. In current prac¬ 
tice compliance with such requirements can be monitored 
by making use of features of advanced operating systems. 
As a result of the standardization the requirements for such 
controls may become sufficiently common that it may be 
profitable to explore architectural features that can simplify 
the demands placed on the operating system. In the quality 
assurance area there is an existing Military Specification on 
Software Quality Assurance Programs that has no apparent 
impact on computer architecture. 13 

Test documentation standards will be concerned with areas 
such as reporting completeness of test, complete description 
of the test environment, and accurate recording of all changes 
to software or the environment over the usually rather ex¬ 
tended period required for test of critical software programs. 
One of the methods for reporting completeness of test, but 
probably not the only one that would be acceptable under 
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such a standard, is the recording of all node exits taken during 
a series of test runs. In current practice this can be done by 
use of staic analyzers to locate the nodes, and subsequent 
“instrumentation” of the nodes such that passage during 
execution can be counted. Execution time of such an in¬ 
strumented program can be orders of magnitude longer than 
that of the original code, and the need to insert and then to 
remove the instrumentation may impair validation of the test. 
This is one of the major obstacles to more widespread use 
of these software test tools, and architectural features could 
possibly be brought to bear to simplify instrumentation re¬ 
quirements and to reduce the execution time penalties. 

STANDARDS FOR SOFTWARE DELIVERABLES 

Software can be delivered to the user in a large number 
of ways: installed in a specified computer, as a listing, as card 
decks, as paper tape, and as magnetic tape to a number of 
formats. Within most of these there are further variations on 
character set, density, and other format variables. Physical 
dimensions of the deliverables and format features are, in 
many cases, covered by standards or trade practices. In most 
cases these affect the design of computer peripherals much 
more than computer architecture proper. As a matter of fact, 
there Seems to be a tendency, through the use of micropro¬ 
cessors, to make peripherals “smarter” (i.e., to adapt them¬ 
selves to format variations) and thus to remove the need for 
some of the conversion tasks that must now be handled by 
processing in the central computer. 

Software documentation is an important deliverable that 
is covered by a number of voluntary standards. 14-15 In ad¬ 
dition, this area is the subject of numerous Department of 
Defense specification and data item descriptions (DIDs). 
None of these documentation standards appears to have an 
immediate or even remote impact on computer architecture. 

CONCLUSIONS 

The imposition of software standards will, in several areas, 
require longer code (i.e., more storage) and slower execution 
(i.e., require a faster computer in order to handle a given 
task). Thus, software standardization imposes a requirement 
for more powerful computers. It would, however, be prob¬ 
ably more correct to say that the capability of industry to 


provide more powerful computers permits standardization 
rather than the other way around. 

The requirements for HOL processing, in particular the 
requirements of newer HOLs, may favor certain architec¬ 
tural features in the central processing unit, and the fact that 
these languages are standardized may impose a further ur¬ 
gency on the computer community to adopt architectural 
features that support these languages. 

Software engineering standards may become accepted in 
the near future. These will require some code expansion, 
control of software access during critical test phases, and may 
lead to more widespread usage of automated test tools to 
quantify completeness of test. Requirements likely to be 
formalized by these standards are currently being met by 
software but hardware features to support the software or 
to supplant it may at some future time be worth investigat¬ 
ing. 

Standards for software deliverables primarily affect pe¬ 
ripherals. Some unburdening of the central processing unit 
by “smart” peripherals may be in the offing. Standardiza¬ 
tion of output formats may further simplify the central proc¬ 
essing requirements in servicing peripherals. 
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INTRODUCTION 

The concept of standardization and the advantages which it 
offered by-passed the semiconductor memory industry for 
the first five years of its existence. Standardization has never 
been easy to achieve in the modern electronics industry, but 
it has been doubly difficult for the fledgling semiconductor 
memory industry. 

This difficulty can probably be attributed to the highly 
competitive nature of the industry combined with the high 
technology levels involved in the products. Each company 
had what they felt were advanced proprietary concepts, 
processes, and designs. This, they felt, would give them a 
competitive edge so they were reluctant to talk to the com¬ 
petition regarding standards, for fear that they might lose 
some of their commercial advantage. 

The lack of standardization extended over a broad range 
of aspects of the industry. The language used by each com¬ 
pany in describing its product was different from that used 
by its competitors, even when the products were intended 
to be interchangeable. 

Beyond the lack of communications standards, there was 
the more fundamental problem that the product images and 
characteristics were not standardized. At one time there 
were over four different IK dynamic MOS RAMS. The 
situation got worse with the advent of the 4K dynamic RAM. 
At least seven different noninterchangeable designs were 
announced, each described in a different language. 

The first 4K devices to be introduced were in a 22 pin 
package. The twelve address bits were supplied in parallel. 
From a performance standpoint, the parts were generally 
compatible. However, the two companies which led the field 
chose to use different pinouts, making the parts incom¬ 
patible. In addition, a third company introduced a part which 
was compatible with one of these except that the output was 
different. It had a low level constant current rather than a 
TLL compatible voltage out. 

Following the original 22 pin devices, there were three 
different 18 and 16 pin devices introduced. The two 18 pin 
devices used different pin-outs and interface logic. The 16 
pin design used a two clock, multiplexed address interface 
logic. 

In addition to the gross differences of pin-out and logic, 
there were numerous cases of subtle differences in timing 


requirements and interface signal levels which created in¬ 
compatibilities between parts. 

Another aspect of Semiconductor Memory where we have 
been hampered by the lack of standards is related to device 
test. A critical aspect of testing is the test pattern used. This 
is the address and data sequence which when combined with 
the power supply voltages and interface voltages and timing, 
are used to determine if a device meets specifications or not. 
Often, in marginal devices, minor variations in the test pat¬ 
tern can make the difference between detecting a bad part 
or not. It is essential that we be able to communicate the 
exact details of these test patterns so that tests may be 
duplicated by vendors and customers. 

Several years ago, a group of engineers representing both 
vendors and users recognized the problems which resulted 
from this lack of standards and set off to correct the situa¬ 
tion. They formed a committee and then petitioned the IEEE 
Computer Standards Committee to sponsor their work. The 
group became formally recognized as the “Semiconductor 
Memory Standards” sub-committee. The membership is 
made up of representatives from device manufacturers, 
manufacturers of computers and related equipment, and 
memory test equipment manufacturers. 

The group normally meets twice a year at the solid state 
circuits conference in February and at the semiconductor 
memory test symposium in October. 

The sub-committee is made up of several task groups 
which meet more frequently to prepare drafts of standards 
to present to the sub-committee. At the present time there 
are three task groups working on standards. The first of 
these is working on standards related to the electrical de¬ 
scription of Semiconductor Memory devices. The second 
group has addressed the problem of describing the test pat¬ 
tern used to test Semiconductor Memory devices. The third 
one, which is now being organized, will develop technical 
standards for describing and measuring the thermal prop¬ 
erties of memories and other integrated circuits. 


STANDARDS IN PROCESS 

There are three standards currently in various stages of 
preparation. The first one is being developed by the device 
description task group: It is entitled “Proposed Standard for 
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Semiconductor Memory Data Sheet Generation”. The intent 
of this standard is to provide a common language and format 
for the industry to use in describing the electrical and me¬ 
chanical properties of Semiconductor Memory devices. 

The document is divided into four sections. The first sec¬ 
tion title ‘‘Product Description” lists a series of different 
items of a general nature which should be included in the 
specification of any product. It also gives some definitions 
and logic conventions which are recommended to be used 
so that the data sheets and specifications can easily be 
understood by everyone. 

Section two, “Product Specification” gives recommen¬ 
dations for the actual electrical and mechanical specifica¬ 
tions of the device. In the various sections, ground rules, 
definitions, and conventions to be used are given. For ex¬ 
ample, it contains a statement which gives a common defi¬ 
nition of the basis for generating the “Absolute Maximum 
Ratings” section. In the past, there was never a definition 
of what this means. Every company had its own ground 
rules for generating these numbers, or in some cases, no 
ground rules. As a result, the numbers are often arbitrarily 
chosen and therefore meaningless. 

In other areas, definitions and conventions are given to 
standardize symbology and the presentation of the data. The 
section on timing specifications is one which has suffered 
from the largest differences from vendor to vendor and as 
a result, the greatest amount of confusion amongst the users. 
The timing specification of current dynamic MOS memory 
devices is extremely complex. There are typically seven 
different signals or groups of signals, each of which have up 
to four critical timing events. Each of these events must be 
specified with respect to at least one other signal and often 
with respect to three or four others. The result is a timing 
specification which often contains a table of over 30 param¬ 
eters with either a minimum or a maximum value, and some¬ 
times both. For this standard, the task group has invented 
a symbology and procedure for generating timing parameters 
to relate any timing event to any other timing event. The 
result is a uniform, nonambiguous “language” for the timing 
specification. A summary of the new symbol procedure is 
included in Appendix A. 

In a similar, though less complex way, the voltage and 
current portions of the specification are covered. Preferred 
symbols are given for the different lines as a function of 
their use within the device. 

The third section covers the area of use related informa¬ 
tion. It includes recommendations for the inclusion of infor¬ 
mation which will simplify the users task of applying the 
device. The items covered are: graphs showing the inter¬ 
action between operating variables, current waveforms, bit 
maps, equivalent interface circuits, testing information, and 
application information. 

The standard is concluded with a series of appendices 
which give: (1) a summary of input/output pin names, their 
symbolism and definitions, (2) a format and example of 
timing specification for a part using the conventions of the 
standard, and (3) a summary of the timing parameter symbol 
procedure. 

The second standard being worked on is one relating to 


the description of test patterns for RAM Memory. The orig¬ 
inal intent of the task group was to accomplish two tasks. 
The first was to develop a language which could be used to 
uniquely define the address and data sequence used in a 
memory test pattern. The second was to use the results of 
the first task to stabilize the definition of some of the more 
commonly used patterns such as march, galpat and walking 
1 / 0 . 

One of the original ground rules for the language devel¬ 
opment was that it should be machine independent and not 
rely on the readers knowledge of the programming language 
or architecture of any of the currently popular memory test¬ 
ers. The reason for this is the wide disparity between tester 
pattern generators used. Some work with an absolute ad¬ 
dress scheme and others use relative addressing. Some de¬ 
fine data as a simple sequential string of bits, while others 
define data as a spacial function of the addresses, while at 
least one has the capability of doing both. 

In its initial efforts, the task group attempted to apply one 
of the more widely understood high level programming lan¬ 
guages, such as APL or Basic. As the standard went through 
successive drafts, the language evolved into one which tends 
to be closer to the language of some of the testers. It is 
undergoing further refinements to satisfy the objections of 
some of the more hardware oriented sub-committee mem¬ 
bers. 

When the language is stabilized, the task group will then 
start to develop a repertoire of test patterns which are in 
common use in the industry. This is being done, not to give 
support to any patterns as more effective than others, but 
to improve communications so that when someone uses a 
pattern name, everyone knows exactly what he means. 

The third standard is one which is concerned with the 
thermal resistance of an integrated circuit. In the past, the 
terms, symbols, and definitions for I.C. thermal resistance 
have not been standardized at all. Each vendor and user tends 
to have his own private set. In addition, some of the defi¬ 
nitions used by device manufacturers have no real value to 
the user in the environment in which he uses the parts. 

The work has progressed to the point of having a first 
draft which has been circulated to the task group for dis¬ 
cussion. It will be at least late 1978 before it has been refined 
to the point where it is ready for presentation to the sub¬ 
committee. 

The draft standard starts with a series of thermal resist¬ 
ance definitions. The list includes the traditional parameters 
R Wc and Re ja , which are favored by the device manufactur¬ 
ers. It also contains several new ones which allow the user 
to predict or measure the I.C. junction temperatures under 
system operating conditions. 

The definitions are followed by a section on test methods 
for measuring thermal resistance. It includes discussions on 
test fixtures, test devices, test instruments and procedures. 

The end result of this effort will be an all inclusive stand¬ 
ard which serves the needs of device designers, manufac¬ 
turers, and users. At the present time, there is no applicable 
standard for anyone, only a variety of sometimes conflicting 
traditions. 

The work of the sub-committee will continue until those 
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topics which are the proper province of the IEEE have been 
considered. 

OTHER STANDARDS WORK 

There is another class of standards which is badly needed 
by the industry but which is not properly handled by an 
organization like the IEEE. This is the standardization of 
the interface image and specifications of the memory devices 
themselves. It was this lack of standards which allowed four 
different IK devices and seven different 4K devices to be 
introduced into the market. There is, however, a committee 
which has been formed, to address this problem. This com¬ 
mittee was formed by JEDEC (Joint Electron Devices En¬ 
gineering Council) as committee JC-42. This committee is 
made up of representatives of device manufacturers only, 
however, there is close communciation between the IEEE 
Semiconductor Memory Standards sub-committee and JC- 
42 so the users voice is well represented. 

In addition to the work of JC-42, there are some additional 
standards work being pursued by other groups. The JEDEC 
committee JC-11 is preparing standards on the mechanical 
packages used to house memory devices. Some of the 
smaller “chip carrier” packages which are now being con¬ 
sidered by this committee can have a long range effect on 
memory architecture. This can come about as a result of the 
increased packaging density which the use of these packages 
will allow. In addition, their use opens up the possibility of 
cost effective multi-chip packages with higher levels of com¬ 
plexity. 


There are numerous standards being generated by the 
N. B. S. or through the MIL specification channels which 
relate to the Semiconductor Memory field. They are broad 
based standards which are intended to cover the general 
field of intergrated circuits. One of these is worth noting 
since it covers topics being considered in the IEEE thermal 
resistance standard. This work, done by G.E., has been 
published in the report RADC-TR-77-321, “Thermal Resist¬ 
ance of Microelectronic Packages.” 

This is an excellent piece of work which covers in a broad 
way definitions and test methods relating to package thermal 
resistance. The work of the IEEE committee is an expansion 
of this work to make it more broadly useful to the commer- 
ical users of integrated circuits. 


CONCLUSION 

The Semiconductor Memory industry has suffered for its lack 
of standards. The result has been a needless waste of re¬ 
sources, both by the vendors as well as the users. The 
progress of the industry has been retarded in ways which it 
is impossible to quantify. The task of making up this lack 
has now been undertaken by two separate groups: (1) an 
IEEE Committee working on standards related to commu¬ 
nication and definitions, and (2) A JEDEC Committee work¬ 
ing to standardize product definitions and images. The com¬ 
bined results of these committees will greatly assist the 
Semiconductor Memory industry to develop new, better 
products and the users to apply them more effectively. 
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This data sheet uses 3 new type of specification nomenclature 
that is derived from background work of several users and 
manufacturers of semiconductor memories. It should help to 
clarify signal and parameter symbols and definitions and make 
data sheet information more consistent. 

ELECTRICAL PARAMETER ABBREVIATIONS 

All abbreviations use upper case letters with no subscripts. The 
initial symbol is one of these four characters: 

V (Voltage) 

I (Current) 

P (Power) 

C (Capacitance) 

The second letter specifies input (I) or output (0), and the 
third letter indicates the high (H), low (L) or off (Z) state of 
the pin during measurement. Examples: 

VOH Output high voltage 

ML = Input low current 

IOZ = Output off current (leakage) 

TIMING PARAMETER ABBREVIATIONS 

AH timing abbreviations use upper case characters with no sub¬ 
scripts. The initial character is always T and is followed by four 
descriptors. These characters specify two signal points arranged 
in a 'frorn-to' sequence that define a timing interval. The two 
descriptors for each signal point specify the signal name and 
the signal transitions. Thus the format is: 


SYMBOLS AND ABBREVIATIONS 

ition nomenclature The signal definitions used in this data sheet are: 


signal name from which interval is defined 
transition direction for first signal 
signal name to which interval is defined 
transition direction for second signal 


T X X X X 

Jit! 


Example: 




A = Address 
D = Data In 
Q = Data Out 
W = Write Enable 
E = Chip Enable 

The transition definitions used in this data sheet are: 

H = transition to high 
L * transition to low 
V ** transition to valid 
X = transition to invalid or don't care 
Z = transition to off (high impedance) 


TIMING LIMITS 

The table of timing values shows either a minimum or a maxi¬ 
mum limit for each parameter. Input requirements are speci¬ 
fied from the external system point of view. Thus, address set¬ 
up time is shown as a minimum since the system must supply at 
least that much time (even though most devices do not require 
it). On the other hand, responses from the memory are spec¬ 
ified from the device point of view. Thus, the access time is 
shown as a maximum since the device never provides data later 
than that time. 

WAVEFORMS 


WAVEFORM INPUT 
SYMBOL 


m 


CHANGE 

WIU CHANGE 

FROM H TO L 

FROM H TO L 

CHANGE 

WILL CHANGE 

FROM L TO H 

FROM L TO H 

DON'T CARE: 

CHANGING: 

ANY CHANGE 

STATE 

PERMITTED 

UNKNOWN 


The drawing shows the address setup time defined as TAVEL, 
Address Valid to Enable Low time. 
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Microprocessor standards 


by TOM PITTMAN 

ltty Bitty Computers and Homebrew Computer Club 

San Jose, California 

and 

ROBERT G. STEWART 

Stewart Research Enterprises 

Los Altos, California 

Now that microprocessors have been with us for three or 
four years, we can stand back and observe some disadvan¬ 
tages of our free enterprise system, as well as obviously 
great advantages. The disadvantages tend to fall most heav¬ 
ily on the shoulders of the user, not the manufacturer. The 
proliferation of microprocessors with incompatible instruc¬ 
tion sets, adds unnecessary chores to learning new mne¬ 
monics, and requires rewriting (at large expense) existing 
software. Another interesting situation can be seen in the 
case of the 8080 and the Z80 processors whose mnemonics 
differ for precisely the same instructions. The act of copy¬ 
righting mnemonics by microprocessor manufacturers places 
the problems in the user’s laps in order to protect the pro¬ 
prietary interests of the manufacturer of the chip. 

Whenever standards are proposed, two reactions are 
voiced, polarizing the opinions. On the one hand we hear 
general disdain: “Who needs them?” or “Standards impede 
progress.” On the other hand we hear cheers: “What took 
so long?” or “Lack of standards impedes progress.” The 
field of microprocessors is not immune to this division. 

We will have standards in software as well as in hardware. 
Unlike the larger mainframe computers, where the entry 
costs are high enough to insure a virtual monopoly on soft¬ 
ware and interface standards by a single manufacturer, there 
are a half dozen or more significant contributors to micro¬ 
processor software for each chip type, and as yet none of 
their products are compatible with each other. There are 
hundreds of components designed for very few processors, 
and many of these cannot even coexist in the same system. 
Standards are inevitable because the users will not tolerate 
this chaos very much longer. 

The absence of standards for the large, batch computers 
today is simply due to the fact that the manufacturers fight 
standardization. Plus compatible peripherals are a touchy 
example. It is not in the mainframe companies interest to 
foster a standard interface. This reluctance to standardize 
extends to software aspects which directly affect a user such 
as job control streams, editor commands, disk I/O calls, tape 
formats, internal data representation, ASCII vs. EBCDIC, 
and pairing of source and relocatable files. These differences 


are particularly vexing for someone who must use computer 
systems made by a number of different companies. If there 
were only one computer company, then standardization ef¬ 
forts would not be necessary. The interesting thing is that 
the present situation looks like it results from a condition 
where all the different computer companies believe that they 
are the only one! Note that most of the problem areas which 
would benefit from standardization would have relatively 
little or no impact on the computer architecture! 

The fun aspects of the hobby or homebrew computer run 
headlong into the software and hardware idiosyncracies of 
the different manufacturers. For example, because the Port 
designations for the console and the console status flag’s 
ready bits have not been standardized, the act of bringing 



"We can't do business after all... my computer isn't 
speaking to your computer." 
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up a system is rendered much harder since a friend with a 
working system may not be able to substitute his proven 
software without first patching in the required changes. This 
patching can be quite difficult because the hardware may be 
flaky also. 

Note that these types of problems are seldom serious to the 
manufacturer, but represent hurdles to the novice user that 
make him curse the people responsible for making life dif¬ 
ficult for him. And typically those people are cloistered 
within companies and make arbitrary decisions without re¬ 
gard to the problems that result for users interfacing with 
hardware and software from a number of sources. 

A veritable jungle has come into existence in the home¬ 
brew or personal computer hobby. Memory boards and 
DMA devices for the S-100 bus are an example. The man¬ 
ufacturers state that they are S-100 compatible, and indeed 
they are unto themselves. However memory boards from 
different manufacturers are, in some cases, not compatible 
with each other. The initial MITS description of the Altair 
or S-100 Bus did not nail down the timing of the bus. By 
this absence, the timing defined by Intel for the 8080 pro¬ 
cessor must intrinsically be recognized as the original timing 
standard for that bus. Later, other processors such as the 
Z-80 and the 2650, have been used on the bus, without 
having the same timing signals and relationship as the 8080, 
so that memory and DMA devices are not directly inter¬ 
changeable. 

Usually it is the user who takes the lumps in both cost 
and frustration because the manufacturer had stated the 
boards are bus compatible, when they actually were not 
because of their timing and control characteristics. 

The IEEE Computer Society Microprocessor Standards 
Committee was formed in the summer of 1977 to try and 
resolve some of the obvious problems that have arisen in 



the use and applications of these marvelous chips called 
microprocessors. The veritable complete revolution in tech¬ 
nique which the microprocessor has brought to the electrical 
engineering profession will be aided for years to come by a 
codification attempt at this time. 

SOFTWARE STANDARDIZATION EFFORTS 

The IEEE Computer Society has taken a bold step for¬ 
ward in forming a standards committee to deal with the most 
pressing of these issues. We have selected three specific 
areas of concern in the software area where immediate ac¬ 
tion can be of tremendous benefit in the microprocessor 
community: 

1. Floating point format and algorithms —We expect to 
be soliciting comments on a first draft of a proposed 
standard by the time of NCC. In this standard we will 
be defining forms for both normal floating point oper¬ 
ations and for extended range and/or precision. Over¬ 
flow and underflow conditions will be specified, and 
minimal requirements of the algorithms implementing 
these operations will be defined. 

2. Assembly language syntax and lexicography —There 
are major questions subject to compromise in this area. 
It is expected that this standard will define the assem¬ 
bler syntax for each of the more popular microproces¬ 
sors, specifying both mnemonics and operand coding. 
By considering all of the processors at one time it is 
hoped that the similarities between processors will be 
reflected in the mnemonics and operand syntax, and 
that guidelines may be developed for the assignment of 
mnemonics to new microprocessors as they appear. In 
particular, it is hoped that assemblers supported by 
timesharing services will all accept the same source 
code for the same chips (this is not now that case), and 
that their reduced development costs in bringing up 
additional processors will make wider support for new 
CPUs more attractive. 

3. Relocatable object module formats —This area, though 
of vital importance, seems to be moving the slowest. 
Again, the major thrust of our efforts is the specifica¬ 
tion of an adequate object module format for each of 
the major processor families. 

Here also it is expected that the careful selection of func¬ 
tional and lexical characteristics will insure a uniformity 
across processor lines which will therefore also extend into 
future CPU design. Clearly the burdens placed on the relo¬ 
cating linker are somewhat arbitrary, so part of our work 
includes the analysis of the kinds and extent of the tasks 
which should be deferred to that phase of program transla¬ 
tion. Obvious candidates for support are the output of macro 
assemblers (see standards effort #2 above) and of FOR¬ 
TRAN compilers. The requirements of other high level lan¬ 
guage compilers should also be considered, as well as the 
possibility of a proper subset (with reduced functions) for 
minimal operating systems. 
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One of the frequent objections we hear points to the dis¬ 
similarities between different microprocessors, with partic¬ 
ular reference to the one-chip controllers. Since most of 
these are single-sourced and lack the multiple-manufacturer 
proliferation of software development tools seen in the 8-bit 
processors like the 8080 and the 6800, we have not been 
overly concerned about extending compatibility in these di¬ 
rections. However, most of the committee members have 
had extensive experience with several microprocessors, and 
our emphasis in each of three areas is focused on the simi¬ 
larities rather than the differences between the varieties of 
chips under consideration. 

HARDWARE STANDARDIZATION EFFORTS 

The initial hardware problems tackled by the Micropro¬ 
cessor Standards Committee are standardization of several 
of the existing and proposed busses, namely the S-100 bus, 
the Intel MDS Bus, National’s Microbus, and the PI bus 
proposed by Pietsch and Khalsa. After the bus issues have 
crystallized, the Committee will look at problems in the 
Small Business and Hobby Computer area, such as inter¬ 
facing and protocols, standards for memory peripherals, 
data formats, disk formats, etc. 

The key issues that arise in standardizing the de facto 
busses are timing diagrams, DMA control, defining unused 
pins for future address, data space, and control extensions, 
and accommodating the higher clock rates of future proces¬ 
sors. The proposed standards for the busses will be given at 
NCC and the audience will have the opportunity to make 
comments on them. They will also be able to make sugges¬ 
tions for future standards activity by the committee. 

Of the existing de facto busses for microprocessors, the 
most widely used is the bus MITS used on its Altair 8800 
computer announced by Ed Roberts in Popular Electronics 
magazine in February, 1975. (Many computer hobbyists re¬ 
gard that as the world’s first digital computer.) The wide 
variety of plug-in boards that are available for the Altair of 
S-100, at competitive prices, have enabled the bus to serve 
the hobbyist and the small business market reasonably sat¬ 
isfactorily. The problems that have arisen with the S-100 
bus are: (1) The use of positive true rather than negative 
true logic, (2) an overabundance of control and non-essential 
signals on the bus, (3) the use of separate data in and data 
out busses, (4) the use of microprocessors with different 
timing characteristics than the 8080, (5) different voltage 
power lines directly adjacent to each other, (6) one common 
AC and DC ground, and to some extent, (7) poorly designed 
boards which plug into the bus. 

The efforts of the IEEE Computer Society Microproces¬ 
sor Standards Committee to resolve some of the S-100 bus 
problems have been spearheaded by Howard Fullmer and 
George Morrow. The timing and DMA protocols adopted 
define processor-memory interactions in a manner which is 
more tolerant of non-8080 processors. All timing relation¬ 
ships are specified only with respect to the phase two (<£2) 
clock signal. Any device proclaiming itself to be a bus mas¬ 
ter, such as a DMA device as well as a processor must 


generate a set of 18 control signals as well as the address 
and data signals. Some DMA devices have failed to generate 
the full set of bus control signals and so have led to incom¬ 
patibility problems with different memory boards which 
were relying on the presence of those signals. The 8224 
system controller can be used by DMA devices and non- 
8080 processors to generate the control signals needed on 
the S-100 Bus. The adoption of standards for the bus will 
enhance its utility in the years to come and help preserve 
the investment which tens of thousands of hobbyists have 
made in their equipment. 

The capability of the chip manufacturers to develop ROM 
chips to store specialized software also will act to force 
standards on higher level languages as they relate to micro¬ 
processors, development systems, operating systems, and 
the personal computing market. Such chips will provide 
BASIC, FORTRAN, PASCAL etc. interpreters and com¬ 
pilers at the selection of the user. Thus the advance in 
hardware technology acts to force upon us the standardiza¬ 
tion of relocatable code formats discussed earlier. 

Packaging standardization for microprocessors has 
evolved to the predominant use of a 40 pin package. As 
longer word length processors are created, more pins may 
well be required unless extensive bus multiplexing is used. 
There is some tendency now to use pins on opposite diag¬ 
onals for ground (pin 20) and power supply (pin 40); how¬ 
ever, even that very limited pin-out standardization is not 
at all universal. Chip designers lay out the chip with all the 
logic capability they can, and only worry about the package 
pin-out at the end. If a pin-out standard is imposed for 
microprocessors, then the chip designer would have to con¬ 
sider the pin-out in the initial chip design, and may have to 
run traces for longer lengths than otherwise in order to reach 
the pads. This acts to reduce the available chip area for the 
device’s logic functions. Thus there are significant semicon¬ 
ductor chip layout and utilization reasons to deter the for¬ 
mulation of a pin-out standard of /uP packages. 

A far more significant aspect of microprocessor architec¬ 
ture and design which would benefit from standardization 
right now is the “timing architecture,’’ or precise sequence 
of signals that interface with external busses and support 
chips. In the practical sense of being able to use existing 
support chips and busses for a new or different micropro¬ 
cessor, the timing architecture is as (or more) important a 
conceptual issue as the register structure and logic capabil¬ 
ity. The dominant existing de facto busses have intrinsically 
been designed around the timing convention of the 8080. A 
timing standard for handshaking and control signals of mi¬ 
croprocessors would be very useful in furthering the uni¬ 
versality of busses, support chips, DMA devices, and con¬ 
trollers. Such a standard should have to be processor 
independent if at all possible. The work which our commit¬ 
tee has undertaken on the S-100 bus timing is essentially 
committed to the same goals, and will probably be helpful 
in developing a standard timing architecture for micropro¬ 
cessors themselves. The committee has attempted to choose 
specific tasks which do not impact architecture in the usual 
sense, and the microprocessor manufacturers in Silicon Val- 
ey agree that our tasks would not. 
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The efforts by U.S. governmental agencies to impose 
standards on computers has resulted in the selection of a 


subset of existing architectures as being acceptable for fu¬ 
ture DOD procurement. Further, the instruction set of the 
AN/UYK 20 minicomputer is being established as the stand¬ 
ard required instruction set for all future minicomputers. In 
addition, some governmental agency staffs active in stand¬ 
ardization efforts believe that a standard bit pattern for the 
processor word is achievable for a given instruction. If such 
a requirement is imposed in the future, then the specific 
processor design and architecture will certainly be impacted. 
An instruction set standard can be dealt with by an assem¬ 
bler or cross-assembler; imposing a standard bit pattern in 
the word would require hardware or a chip design that would 
implement the standard. Whether the conventions that have 
been established by DOD for computers and minicomputers 
are forced downward on microprocessors remains to be 
seen. 

The rapid changes in technology have led to multiple 
sources of software and hardware to create a system, as 
compared to a single source, bundled situation which has 
prevailed in the past. For the user to benefit from the re¬ 
markable advances in technological capability, it really is 
essential to formulate suitable software and hardware stand¬ 
ards now. 
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Reflections in a pool of processors— 

An experience report on C.mmp/Hydra* 


by WILLIAM A. WULF and SAMUEL P. HARBISON 
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INTRODUCTION 

This paper is a frankly subjective reflection upon the suc¬ 
cesses and failures in a large research project—the construc¬ 
tion of a multiprocessor computer, C.mmp, and its operating 
system, Hydra—by those most intimately involved in its 
design, construction, and use. 

C.mmp and Hydra have now reached a sufficient level of 
maturity to establish themselves as useful and reliable com¬ 
puting resources at Carnegie-Mellon University. The user 
community has grown from primarily operating system im¬ 
plementors to include researchers in other operating systems 
and multiprocessors and casual or curious users interested 
in using the unique features of the system (e.g., the Algol 68 
language, whose first implementation at CMU was on 
C.mmp.). 

Some of the scientific results we originally hoped for have 
been published and are listed in the bibliography at the end 
of the paper. Other results will be published in the future as 
we observe the system under varied loads and over longer 
periods of time. In addition to these factual results, however, 
we have learned a number of things of a more subjective 
nature—things that we did right and, perhaps more impor¬ 
tantly, things that we did wrong. We believe that many of 
these lessons are not unique to our project, and their pres¬ 
entation here will be valuable to the larger computer science 
community. 

For those people unfamiliar with C.mmp and Hydra, we 
shall provide a brief overview of multiprocessor research at 
CMU, and some details about C.mmp, Hydra, and the goals 
we originally set for the research project. This information 
should serve as a general background against which our 
evaluation of the project can be cast. The interested reader 
will find more details in the bibliography. 

Multiprocessor research at CMU 

In late 1971 we at CMU decided to embark on a research 
program to explore multicomputer structures—especially 
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those structures in which the several computers share a 
common address space. At the time it appeared to us that 
the economics of LSI technology would make multi-mini or 
multi-micro structures the architecture of choice for many 
medium- to large-scale applications. In addition to the eco¬ 
nomic arguments, there appeared to be many other advan¬ 
tages to such structures, including high availability, expans- 
ability, and so on. 

Despite the fact that a number of multiprocessor com¬ 
puters had been built prior to 1971, relatively little of a 
scientific nature was known about them. Our goal was to 
explore a number of alternative multiprocessor designs, ex¬ 
amining both the hardware and software issues, and to re¬ 
port on these explorations. To that end we undertook the 
design and construction of two multiprocessor systems, 
C.mmp and Cm*, and their associated software. 

C.mmp, the subject of this paper, is a relatively straight¬ 
forward multiprocessor. Begun in 1972, it connects 16 pro¬ 
cessors to a large shared memory (up to 32 megabytes) 
through a central crosspoint switch. The access time from 
any processor to any word of memory is identical. Cm*, 
started in 1975, replaces the crosspoint switch with a dis¬ 
tributed, bus-oriented interconnection scheme between pro¬ 
cessor-memory pairs. In contrast to C.mmp, the access time 
from a Cm* processor to a word of memory can vary by an 
order of magnitude depending upon the particular processor 
and memory module involved. These two machines have 
quite different implications on the software which runs on 
them; between them we are able to explore many of the 
interesting issues of distributed processing. 


C.mmp 

C.mmp is a multiprocessor composed of 16 PDP-ll’s, 16 
independent memory banks, a crosspoint switch which per¬ 
mits any processor to access any memory, and a typical 
complement of I/O equipment. A path through the switch is 
independently established for each memory request and up 
to 16 paths may exist simultaneously. An independent bus, 
the IP-bus, carries control signals from one processor to 
another; no data is carried by this bus. Collectively the 16 
processors execute about 6 million instructions per second; 
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the total memory bandwidth is about 500 million bits per 
second. In short, despite the fact that it is built from mini¬ 
computers, C.mmp is a large-scale machine. 

The current configuration of C.mmp includes 5 PDP-11/ 
20 processors (5 usec/instruction), 11 PDP-11/40 processors 
(2.5 usec/instruction), and 3 megabytes of shared memory 
(650 nsec core and 300 nsec semiconductor). All of the 11/ 
40 processors have been modified to include writable mi¬ 
crostores; thus we are able to tailor their instruction sets to 
specific applications. The cost of this configuration is 
roughly $600,000, of which $300,000 is the cost of proces¬ 
sors, $200,000 is memory, and $100,000 is the switch, IP- 
bus and other special equipment. Of course, there is an 
additional cost associated with I/O devices. 


Hydra 

Hydra is the “kernel” of the operating system for C.mmp; 
it is not intended to provide most of the familiar features of 
an operating system (e.g., it does not provide files, a com¬ 
mand language, or even a scheduler). Rather, Hydra pro¬ 
vides an environment in which it is (intended to be) easy to 
write user-level programs that supply these familiar facili¬ 
ties. Hydra was designed in this kernal fashion in order to 
permit (and encourage) experimentation with features and 
policies appropriate to multiprocessors. 

Hydra, which was a research project in its own right, uses 
a capability-based protection structure, a scheme in which 
only the possession of the appropriate kind of reference to 
an object (e.g., a file) grants access to that object. In order 
to allow user-level definition of operating system facilities, 
Hydra extends the basic capability scheme with the ability 
to define new types of objects and (protected) operations on 
these object types. Thus it is possible for a user to define 
new types of files, processes, message buffers, or whatever. 
These newly defined types share an equal status with those 
that already exist—which is another way of saying that 
Hydra attempts to preempt as few decisions as possible, 
thus allowing the users to tailor the system to their needs. 

Software already built on top of Hydra in this manner 
includes file systems, directory systems, schedulers, and 
language processors (for Algol 68, L*, and a flexible com¬ 
mand language). 

Project goals 

Two general goals influenced both the hardware and the 
software design from the outset. The C.mmp/Hydra system 
was envisioned as both symmetric and general purpose. By 
symmetric we mean that replicated components, such as 
processors, are treated as an anonymous pool; no one of 
them is special in any sense. By general purpose we simply 
mean that we did not intend to cater to only those programs 
which need a multiprocessor; the multiprocessor character 
of the machine is used to improve throughput across a set 
of independent jobs as well as to multiprocess single jobs. 
Both the hardware and software were designed with these 
goals in mind. 


The symmetry goal is manifest in a number of ways. At 
the hardware level, for example, an interprocessor interrupt 
mechanism was designed so that every processor could in¬ 
terrupt every other processor (including itself) with equal 
ease. At the software level there is no “master-slave” re¬ 
lation among the processors—any processor may execute 
any part of the operating system at any time (subject, of 
course, to mutual exclusion in accessing shared data struc¬ 
tures). At the user level, a job may execute on any proces¬ 
sor, and indeed may switch from one processor to another 
many times during its execution. 

The impact of the general purpose assumption is more 
subtle; it implies that we have to provide a broader range of 
software than would be expected if our focus had been more 
narrow. It also implies that optimizations to a specialized 
problem domain should not be made in the operating system. 
Some of the specific effects of this goal will be found later 
in the evaluations. 


Performance evaluation tools 

Many of our evaluations of C.mmp are based on data 
obtained from a number of tools designed to measure system 
performance. Although not one of our greatest successes, 
we think these tools are important enough to present here. 
We have three measurement tools: a script driver, a hard¬ 
ware monitor, and a kernel tracer. 

The Script Driver is a program which can place a meas¬ 
ured load on the system by simulating a number of users at 
terminals performing various tasks. This known load can 
make the interpretation of performance measurements much 
easier. 

The Hardware Monitor is a device built at CMU which 
can monitor in real time the signals on a PDP-ll’s bus. The 
Monitor is very useful in measuring the activity of a single 
C.mmp processor, and for recording the activity of small 
portions of the operating system. It is less effective in meas¬ 
uring total system performance. 

The Kernel Tracer, the most commonly used tool, is built 
into the Hydra kernel. It allows selected operating system 
events (e.g., blocking on semaphores, context swaps) to be 
recorded while applications are running. The accumulated 
data can be processed off-line to give a detailed record of 
what was happening on each processor. Naturally, the use 
of the tracer slows down the entire system, but this obvious 
point doesn’t really seem to matter in practice. 

The importance of these tools should not be underesti¬ 
mated. In any system as complex as an operating system, 
design decisions are often based on intuitive assumptions of 
performance tradeoffs. Without accurate measurements, 
these design assumptions cannot be verified. Certainly we 
found that some of our assumptions were wrong, causing us 
to redesign several parts of Hydra. 

Format of the paper 

The body of this paper is a highly edited report of a 
meeting called specifically to evaluate the C.mmp/Hydra 
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project. The attendees were representatives of the various 
groups involved in the design, implementation, and use of 
C.mmp and Hydra; hardware designers, operating system 
implementors, those doing performance evaluation, and four 
major users. In all, sixteen persons attended, the maximum 
number we felt could interact productively. 

The purpose of the meeting was to solicit the opinions of 
the participants concerning the nature of our successes and 
failures. We had also solicited written opinions from a wider 
group—in fact, just about everyone who has had anything 
to do with C.mmp and Hydra. The participants knew, of 
course, that the results would be reported in this paper. 

The meeting and written responses produced over a 
hundred distinct comments. To organize these in a coherent 
fashion we asked the participants to decide upon our five 
greatest successes and five greatest failures. With some ex¬ 
ceptions the comments have been organized under these 
headings; the participants’ comments have been indented to 
separate them from background information and summary 
comments. 

Any paper that sets out to reflect upon the successes and 
failures of a research project is potentially self-serving. We 
were extremely conscious of that danger and have at¬ 
tempted, through the format of the meeting and the editing 
of its transcript, to construct the paper in a manner which 
minimizes this effect. Either our initial fear of being self- 
serving was groundless, or the format chosen worked ex¬ 
tremely well. We shall let the readers judge for themselves, 
but we feel that the result has been a reasonably objective, 
well-balanced view of the C.mmp/Hydra project. 


OUR GREATEST SUCCESSES AND FAILURES 

We shall begin this report with what, in fact, happened 
last at the meeting—a listing of our most notable accomplish¬ 
ments and mistakes. This list was created after all opinions 
had been expressed, thus the participants had the opportu¬ 
nity to hear the opinions of the others before deciding upon 
the content of the list. To keep the discussion crisp we 
arbitrarily chose to limit each list to five items. Surprisingly 
(to the editors at least), despite the differing interests of the 
participants there was essentially complete agreement on 
the items to be included on each list. 

Our notable accomplishments: 

We constructed a cost-effective, symmetric multiproces¬ 
sor. 

We provided, in Hydra, a capability-based protection sys¬ 
tem which allows the construction of operating system 
facilities as normal user programs. 

We were able to distribute the Hydra kernel symmetrically 
over all processors. 

We provided successful mechanisms for the detection of, 
and recovery from, software and hardware errors. 

We used an effective methodology for constructing the 
Hydra kernel. 


Our notable disappointments: 

The hardware is less reliable than we would like. 

The small address of the PDP-11 has a large negative 
impact on program structure and performance. 

We are unable to partition C.mmp into disjoint systems. 
We did not put enough human-engineering into the soft¬ 
ware interface to the user. 

We did not give enough attention to project management. 

Neither our successes nor failures are, of course, unqual¬ 
ified, and the story behind each is littered with smaller 
successes and mistakes. Moreover, there are dependencies 
between the things that went well and those that didn’t; the 
fact that we have a running 16-processor system must be 
tempered, for example, by a poorer-than-expected reliability 
record. The reliability record, on the other hand, led us to 
greater concern for software structures that detect and sur¬ 
vive hardware malfunction—and we count those structures 
among our most important accomplishments. For all these 
reasons, while we have used the success/failure list to or¬ 
ganize the paper, one should not expect all the points listed 
under a “success” to be positive in nature. On the contrary, 
we believe it important to expose the contributing events, 
both positive and negative, as well as the major points listed 
here. 

With that introduction then, here is the report of the 
meeting. 

THE SUCCESSES 

A cost-effective multiprocessor 

C.mmp’s design goals included speed, simplicity, and the 
use of as many commercially-available components as pos¬ 
sible. Because C.mmp is a unique computer some critical 
parts had to be designed and built especially for the project. 
While this was a burden, it did give us maximum freedom 
in the design of these critical components, including the 
crosspoint switch, the IP-bus, and the processor modifica¬ 
tions for memory relocation. These were all built by the 
CMU Computer Science Department Engineering Labora¬ 
tory. 

The basic design goals have been justified by experience, 
with speed having been the least important emphasis. 

CMU-built hardware is not a large proportion of the total 
system cost. 

The crosspoint switch is very reliable, and fast enough. 

The use of immediately available components was a major 
factor in getting C.mmp built as fast as we did, but it 
limited us in taking advantage of technology which de¬ 
veloped in succeeding years. 

We were especially happy about the evaluation of the 
crosspoint switch, which many people thought would be 
C.mmp’s Achilles’ heel. In retrospect we think we were too 
concerned about raw speed in the design of the switch and 
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memory; as it turns out, most applications are sped up by 
decomposing their algorithms to use the multiprocessor 
structure, not by executing on a processor with short mem¬ 
ory access times. 

The comments at the meeting did reflect some specific 
complaints about the hardware, several of which we later 
decided were significant enough to be listed as some of our 
major disappointments. Many of these stemmed from our 
choice of a processor for C.mmp. In 1971, only the PDP-11/20 
minicomputer met our requirements. In 1974 we decided to 
take advantage of technology advances and use the new, 
faster PDP-11/40 processors to complete C.mmp. One fea¬ 
ture of the PDP-11 architecture which might be expected to 
impact the goal of symmetry for C.mmp is the close asso¬ 
ciation of an I/O device with exactly one processor. 

The PDP-11 processors required more modifications than 
we expected to ensure the security of the operating sys¬ 
tem. 

The PDP-1 l’s 16-bit address is too small for many inter¬ 
esting applications. 

Having to supporting two PDP-11 models complicated the 
development of the processor modifications and the op¬ 
erating system. It would have been better to have had a 
single processor model, regardless of its speed. 

Having I/O devices bound to particular processors made 
it difficult to move a device from a malfunctioning pro¬ 
cessor to a good one, but device utilization was not oth¬ 
erwise sacrificed. 

Perhaps more than anything else, our experience with the 
PDP-11 has given us a much clearer idea about what features 
are really important in choosing a processor, and which are 
not. Our consensus is that speed is not very important, for 
reasons already cited in conjunction with the crosspoint 
switch. Reliability is very important, but we found that much 
can be done in software to increase the overall system re¬ 
liability, as long as the hardware has some basic error-de¬ 
tection mechanisms. (Our own approach to this is described 
later.) The address size is important because if it is too small 
for the expected applications, the ensuing problems cannot 
be completely overcome by software. The PDP-11 I/O ar¬ 
chitecture is an example of a feature that turned out to be 
unimportant because it could be completely hidden from 
users by software. 

At a higher level, users of C.mmp seemed satisfied with 
the overall system performance. 

Our ability to support multiprocess algorithms is well es¬ 
tablished by the performance of the many applications on 
C.mmp. 

We have successfully supported user processes that re¬ 
quire real-time response, although this was not one of our 
major goals. 

At the end of the paper we will give some performance 
figures for an application which runs on several CMU com¬ 
puters, including C.mmp. 


Most often cited criticisms of the system were: 

Interaction with operating system facilities, in or out of 
the kernel, is accompanied by a high overhead. 

The most serious obstacle to rapid execution of large 
systems is the limitation imposed on programming by the 
small PDP-11 address. 

Memory contention significantly degrades performance 
when many processes are accessing the same memory 
page. This is usually caused by the processes sharing the 
same code pages. 

Memory contention is very serious when using high-per¬ 
formance I/O devices which depend on rapid access to 
memory during transfers. 

The performance bottlenecks are due to a combination of 
avoidable and unavoidable factors. We were initially dis¬ 
tressed at the high operating system overhead (it takes about 
500 microseconds to enter and exit the kernel), but we at¬ 
tribute most of it to a lack of experience with the fairly 
complex features we wished to implement. We are confident 
that the overhead is not an inevitable result of our protection 
mechanisms, nor is it due to the hardware design. 

Memory contention, caused by several processors trying 
to access the same memory simultaneously, was a perform¬ 
ance concern from the outset of the project. Our simulation 
studies indicated that its effect would be minimal, but in 
practice several circumstances conspired to make the prob¬ 
lem significant. First, typical large multiprocess applications 
tend to share the same code among all processes, and this 
greatly increases the probability of accesses to the same 
memory. Second, the installation of per-processor caches, 
which were to handle this code-reference problem, has been 
delayed due to various resource shortages. Finally, we found 
that devices such as our disks and drums could not tolerate 
the long memory access times characteristic of periods of 
high contention. A software solution to this problem had to 
be implemented. 

The small address problem is serious for large applications 
which cannot fit within the 64K address space on the PDP- 
11. Although we could not have avoided this problem, we 
were guilty of underestimating its significance for the appli¬ 
cations which were to run on C.mmp. The problem is con¬ 
sidered in more detail later in this paper. 


Protected subsystems 

In Hydra, the construction of operating system facilities 
outside the kernel is centered around an abstraction called 
a protected subsystem. A subsystem is, in its basic form, a 
new object type combined with a set of procedures which 
operate on objects of that type. 

Our experience derives from over twenty working sub¬ 
systems implementing schedulers {Policy Modules in Hydra 
terminology), files, directories, an I/O device allocator, and 
a host of other traditional operating system facilities. As 
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software development continued by diverse users, we were 
curious to see whether all the required software could be 
built within the subsystem abstraction, whether such devel¬ 
opment could be done easily and quickly, and whether the 
resulting facilities could be easily merged into the user en¬ 
vironment. 

The protected subsystems abstraction is very powerful in 
designing operating system software in a capability envi¬ 
ronment. 

It is easy to design subsystems which are easy to use and 
which are protected from any interference from software 
outside the subsystem. 

The subsystem structure makes it easy to provide several 
coexisting and competing facilities. 

The subsystem structure is useful for isolating facilities 
under development or being debugged. 

New subsystems are easily incorporated into the standard 
system. 

We think the subsystem concept in Hydra is as useful as 
the closely-related notion of extended data types has been 
in the field of programming languages. Part of the original 
motivation for the subsystem concept was our desire to 
allow alternate solutions to problems which we could not 
foresee in a multiprocessor environment. However, we 
found that subsystems are also very useful in debugging 
versions of “standard” systems without interfering with 
users. 

Many people at the meeting were critical of the failure to 
follow up the subsystem design with the software tools 
which would encourage building subsystems in this new 
environment. 

Subsystem construction still suffers from being ad hoc, 
there being inadequate software support for managing the 
programs, data structures, and documentation which com¬ 
prise the subsystem. 

The development of system software (subsystems) by 
many different people makes it more difficult to impose 
any standardization. 

Subsystems are less likely to be successful when they 
attempt to implement traditional (non-capability) systems 
in traditional ways. 

These problems are the result of our not giving the user 
environment outside the kernel as much attention as we 
gave the Hydra kernel itself. We consider it one of our worst 
mistakes and will discuss it more later in the paper. 

Scheduling is an example of a traditional operating system 
function which, in Hydra, is partially implemented outside 
the kernel by a subsystem called the Policy Module (PM). 
We thought that providing scheduling policy outside the 
kernel would allow us to experiment with different special¬ 
ized strategies for scheduling cooperating processes. 


The first Policy Module is a distinguished subsystem for 
several reasons. First, it was one of the first subsystems 
built outside the kernel and exhibits many of the mistakes 
of any first attempt. Second, it is a particularly nice example 
of our ability to build operating system facilities outside the 
kernel. Finally, it interacts very closely with the kernel, so 
the efficiency of the kernel interface is emphasized. 

The first Policy Module was operational from 1974 
through May, 1977. Our basic evaluation at the meeting was 
that 

The first Policy Module adequately demonstrated that tra¬ 
ditional policy decisions could be made outside the kernel. 

In spite of this, many people noted flaws in the imple¬ 
mentation which were glossed over in our rush to see if the 
PM would work. 

Insufficient attention was paid to reliability and through¬ 
put in the Policy Module. 

The PM-kemel interface turned out to be more complex 
than we had anticipated. 

We included things in the kernel facilities which logically 
belonged outside; this acted to complicate the kernel in¬ 
terface. [For efficiency reasons, we implemented in the 
kernel some facilities which should have been outside 
according to our philosophy. 

Hence, 

The construction of Policy Modules was not as easy as 
we had imagined before we actually tried it. 

Because we expected a PM to incorporate specific knowl¬ 
edge about the processes it was scheduling, we anticipated 
having many PM’s simultaneously scheduling different sets 
of processes. Indeed, having several PM’s run at the same 
time was no problem, but again the performance left some¬ 
thing to be desired. 

To support multiple Policy Modules, more facilities are 
needed in the kernel to ensure a fair allocation of proces¬ 
sor and memory resources to each Policy Module. 

We began to build a second version of the Policy Module 
almost as soon as the deficiencies in the first were recog¬ 
nized. This design proceeded in parallel with performance 
improvements to the first PM, and in fact we were running 
both PM’s simultaneously for a short time. 

The distributed operating system 

Hydra was designed with no master-slave relationship 
among processors. With the exception of the lowest level of 
I/O device support, all system tasks may run on any and all 
processors. An immediate result of this is that we expected 
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a high degree of parallelism in Hydra and the corresponding 
need for effective synchronization methods. 

There are two notable aspects to our approach to syn¬ 
chronization. First, we decided to synchronize on data 
rather than code. Every data structure which can be ac¬ 
cessed by more than one processor is provided with a lock 
or semaphore which is used to ensure mutual exclusion. 

Second, we provided a range of synchronization primi¬ 
tives, from very fast “locks” to much slower “sema¬ 
phores.” The tradeoff here is the overhead needed to P or 
V the lock or semaphore against the resources which will 
be tied up by a process waiting to pass the lock or sema¬ 
phore. Small data structures which are locked for short 
periods of time (order 300 microseconds) use locks, which 
involve a very small overhead (approximately four instruc¬ 
tions) when the process does not block. Large data struc¬ 
tures, or data structures whose processing may be inter¬ 
rupted for long periods of time (as when waiting for I/O) use 
semaphores, which tie up fewer resources when blocking is 
necessary. 

The simple, symmetric hardware has permitted a much 
simpler operating system design. 

Hydra hides the processor-device correspondence so well 
that most of Hydra, and all the software at the user level, 
is unaware of the actual location of I/O devices. 

The symmetric distribution of the operating system has 
been an unqualified success. We are able to achieve a 
high degree of parallelism within Hydra, and the system 
is insensitive about the number of processors available. 

The use of asynchronous processes (“demons”) to imple¬ 
ment system functions resulted in simpler designs and 
improved performance. 

In providing synchronization within the kernel, we believe 
we profited by locking data structures rather than code. 

Our decision to provide several types of synchronization 
mechanisms gave us much design flexibility. 

The natural synchronization primitives and our conscious 
and constant commitment to a high degree of parallelism 
has resulted in our encountering few software bugs caused 
by inadequate synchronization. 

We have found that the use of demons to absorb much of 
the system work load outside the normal computational 
stream has simplified much of Hydra’s design. We might 
not have used this technique if we did not have so much 
confidence in our synchronization techniques and our ability 
to achieve a high degree of parallelism. 

Coverage of hardware and software errors 

There are times when clouds do have silver linings. From 
the earliest days of the project we had to contend with 
unreliable hardware and our own software mistakes; more¬ 
over, we could not afford a 24 hour/day operator to reload 


the system after each crash. Thus we were forced to con¬ 
sider the general problems of software detection and recov¬ 
ery from errors—whether they be hardware or software 
induced. 

When an error is detected by Hydra, we try to answer a 
number of questions. What was the exact error? Can we tell 
if it is due to a hardware or software malfunction? If hard¬ 
ware, is the problem repeatable or transient? Have any 
critical data structures been damaged? If so, can the damage 
be repaired? Can we eliminate a piece of malfunctioning (or 
just suspicious) hardware and still run? In all cases, our aim 
is to keep the system running with as much functionality as 
possible. 

Our probability of detecting an error soon after it has 
occurred is increased by building error-detection mecha¬ 
nisms into the hardware and software. The CMU-built mem¬ 
ory relocation units implement parity checking on every 
memory byte and on the address bus through the crosspoint 
switch. Software modules employ redundant representation 
and other techniques to try to limit the propagation of errors 
not detected by the hardware. 

Recovery mechanisms invoked by the detection of any 
error employ a “suspect-monitor” paradigm to ensure that 
a failure in the recovery processor may be detected cleanly. 
Two processors are always involved; one, the suspect, at¬ 
tempts to record the system state at the time of the error; 
the other, the monitor, watches the suspect and assumes 
control if the suspect is unable to finish. The suspect is 
always the processor on which the error occurred. The mon¬ 
itor is selected at random from all other processors. There 
are a number of steps which can be taken during a recovery 
action depending on the type of error, including removing 
processors or memories from the system and producing ex¬ 
tensive crash dumps for later off-line analysis. 

The fault tolerance built into some kernel modules re¬ 
sulted in making them among the most reliable in the 
system—more reliable than other modules coded by the 
same programmer without using such techniques. 

The software facilities for detecting software and hard¬ 
ware errors and restarting the system automatically have 
been a big success. 

Similar facilities in user software are beginning to be de¬ 
veloped and show much promise in improving overall 
system reliability. 

Even though we are proud of our current error-handling 
mechanisms, we know that system needs more work in this 
area, particularly in the area of supplying policies to deter¬ 
mine which mechanisms should be invoked for different 
types of errors. While it is true that we can recover from 
virtually any error by initiating an automatic reloading of 
the operating system, this is a drastic action we would like 
to use only in the case of truly catastrophic errors. Unfor¬ 
tunately, the difficulty in pinpointing the exact location of 
some hardware errors and the difficulty in verifying the 
consistency of the complex capability data structure has 
resulted in our classifying almost all errors as “cata- 
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strophic” in this sense. We are in the midst of redesigning 
both hardware and software to correct these deficiencies. 

Software development methodology 

Our initial goals for the Hydra implementation did not 
explicitly include the notion of exploring a software engi¬ 
neering methodology. Nevertheless, we used a method 
based on Pamas’ “modular decomposition”* and it worked 
quite well; indeed many of us believe that without it the 
project would not have succeeded. 

The methodology used caused us to divide the units of. 
work (programming tasks) along the lines of the major data 
structures in the system. A module (and hence a program¬ 
mer) was responsible for the representation of, and all op¬ 
erations on, a data structure. No one other than the respon¬ 
sible programmer had access to knowledge concerning the 
implementation details. 

Because methodology per se was not our major goal we 
were not fanatical about enforcing the methodology, and 
were often less precise about the specifications than we 
might have been. Both the positive and negative aspects of 
this informal approach are reflected in the following re¬ 
marks: 

We believe that it is a measure of the success of the 
modular implementation of the kernel that one full-time 
programmer can maintain this program which comprises 
2000 (listing) pages of source code. 

The independent implementation of the modules in Hydra 
resulted in a lack of any uniform coding style and in some 
duplicated effort in interfacing to the underlying hardware. 
The effect was not very serious since all the implementors 
were highly talented, exhibiting differences in style rather 
than quality. 

Because modules were implemented independently, no 
one initially had a detailed knowledge of the entire system. 
This made debugging more difficult and resulted in a dif¬ 
ficult transition when Hydra began to be maintained by a 
single programmer who was not part of the original im¬ 
plementation team. 

Coding of the kernel began quickly after the initial design. 
Some think too quickly. 

Loose management coupled with the modularization tech¬ 
nique worked well except in promoting a standardization 
of coding styles. 

Information hiding as a modularization technique resulted 
in coding situations in which information necessary to 
make a decision was not available. 

As Hydra developed and was modified, the original, clean 
modularization began to break down as new features were 
added and performance bottlenecks removed. 


* Pamas, “On the Criteria to be Used in Decomposing Systems into Mod¬ 
ules,” CACM, 15, 12, pp. 1053-1058, 1972. 


We still think the modular decomposition methodology is 
extremely good for structuring large systems. In our expe¬ 
rience, breakdown of the modular structure occurs mainly 
when programmers in the midst of debugging adapt “quick 
and dirty” solutions which do not preserve modularity. 

All but a very small part of Hydra is written in a high- 
level implementation language, Bliss-11. There seems to be 
no question that it was possible, indeed advantageous, to 
write the kernel in Bliss, but there were problems. The Bliss- 
11 compiler was developed only shortly before the kernel 
was begun and was an independent research project (inves¬ 
tigating compiler optimization techniques). There was some 
initial friction between the two groups, but both appear to 
have benefited in the long run. 

The Bliss-11 compiler was designed to compile a slightly 
modified version of the Bliss-10 language into very com¬ 
pact PDP-11 code. This it does. 

The implementors of the Hydra kernel were, and continue 
to be, a major influence on the addition of new features 
to Bliss-11. 

The facilities of the Bliss-11 language and compiler had a 
significant influence on the coding of Hydra. 

Some of us believe that Hydra could not have been written 
in this environment without a language of Bliss’s caliber. 

Bliss-11 preceded Hydra by too short a time. The unreli¬ 
ability of the compiler during its first year of use hindered 
kernel development. 

Compatibility between Bliss-11 and Hydra was a problem. 
Changes in Bliss-11 sometimes had unfortunate conse¬ 
quences on Hydra code. 

We think these comments reflect the close interdepend¬ 
ence between a large programming project (Hydra) and the 
software engineering tools it uses (Bliss-11). Bliss was in a 
real sense critical to Hydra’s development. The need to 
debug both Bliss and Hydra simultaneously was a necessary 
burden. 

A common measure, albeit a crude one, of a methodology 
is the productivity of the programmers which used the meth¬ 
odology. By that measure our development strategy worked 
very well; the average productivity has been about 20 in¬ 
structions per man-day for kernel code (the typical industrial 
average for similar code is 5-7 instructions per man-day). 

THE FAILURES 
Hardware reliability 

Hardware (un)reliability was our largest day-to-day dis¬ 
appointment at the time the evaluation meeting took place. 
The aggregate mean-time-between-failure (MTBF) of 
C.mmp/Hydra fluctuated between two to six hours, where 
a failure is defined to be any situation which triggers the 
recovery actions described earlier. About two-thirds of the 
failures were directly attributable to hardware problems. 
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There is insufficient fault detection built into the hard¬ 
ware. 

We found the PDP-11 UNIBUS to be especially noisy and 
error-prone. 

Our paging drums were chosen for their predicted per¬ 
formance, but their reliability was so poor that perform¬ 
ance was often a moot point. 

The crosspoint switch design is too trusting of other com¬ 
ponents; it can be hung by malfunctioning memories or 
processors. {This almost never happens, but when it does 
automatic recovery is impossible .] 

We made a serious error in not writing good diagnostics 
for the hardware. The software developers should have 
written such programs for the hardware. 

In our experience, diagnostics written by the hardware 
group often did not test components under the type of 
load generated by Hydra, resulting in much finger-pointing 
between groups. Faulty hardware is often kept in the user 
system because only Hydra can provoke and pinpoint 
errors. 

Several components of the system have gone through 
several development cycles, mostly to improve the handling 
of exceptional conditions, but we are basically limited by 
the capabilities of the PDP-11 and its UNIBUS. There ap¬ 
pear to be two flaws in many of the off-the-shelf compo¬ 
nents. One of these was mentioned during the meeting: the 
lack of mutual suspicion. There are a number of ways in 
which the entire system can be made to fail if one inessential 
component does not operate according to specifications. 
The other flaw was not mentioned: the failure to contain 
errors. Once an error has been detected the goal should be 
to make absolutely sure that the damage won’t spread. Many 
of the standard components, unfortunately, will “complete” 
an operation even when an error is known to exist; in com¬ 
pleting the operation they destroy data, thus making the 
error unrecoverable. 

There is some good news to report, however. Following 
the meeting, increased emphasis was given to hardware 
maintenance. As this paper is written (January 1978) our 
MTBF has increased to about ten hours and many of the 
hardware problems seem to be settling out. 


The small address space problem 

The PDP-11 is a 16-bit minicomputer; of particular interest 
is the fact that this restricts all addresses generated by a 
user program to be 16 bits long. These 16 bits can be used 
to address no more than 64K bytes of memory. We refer to 
this limitation as the “small address problem”, or SAP. 

Although we were initially aware that the operating sys¬ 
tem would have to provide some sort of facility for allowing 
a user to address more than this amount of memory, we did 
not appreciate how restrictive the 16-bit limitation would be 
or to what extent circumventing it would affect performance. 


Our initial impression was that the 16-bit limitation would 
be offset by the ability to create multiprocess programs— 
that the typical program organization would be a larger num¬ 
ber of processes, each addressing a smaller amount of mem¬ 
ory. That impression turned out to be false, as is reflected 
in some of the comments made at the meeting: 

Our initial prediction that programs would be implemented 
as small subsystems using less than 64K was wrong. 

Multiprocess algorithms do not always produce small pro¬ 
grams. 

Even though programmers are writing programs which 
execute on PDP-1 l’s, their tasks are CDC 6600-size. 

There is nothing good to say about this problem other 
than that we were pretty much forced into it. 

To circumvent this problem, Hydra provides a facility, 
supported by the hardware, to divide the address space into 
8 pieces, each of which is called a “page.” The user is 
permitted to have an indefinitely large number of pages, but 
to address only 8 of them at any instant. Operating system 
facilities are provided to allow the user to dynamically des¬ 
ignate which of his pages are to be addressable; he does this 
by associating a page with one of the 8 “relocation registers” 
maintained by the hardware. Thus, except that the cost of 
loading is larger, the addressing scheme is very similar to 
the use of “base registers” on 360-370 style machines. We 
have found this facility, however, to be less than ideal. 

Page boundaries are absolute, and the programmer must 
always be aware of them. 

The problem is in addressing data. There are easy solu¬ 
tions to addressing code segments. 

More relocation registers and a smaller page size would 
reduce but not eliminate the problem. 

We believe the problem would exist even if making pages 
addressable required no overhead. 

Because of the performance penalties associated with 
managing the address space, the inconvenience cannot be 
hidden from the user through a high-level language: 

L*’s ability to allow access to large amounts of memory 
has been hindered by the short PDP-11 address. [L* is a 
list processing language used for the implementation of 
large systems.\ 

It must be emphasized that not all programs are affected 
by the small address space problem: 

In practice, most subsystems have no problem fitting into 
64K. 

Our failure on the small address problem was really one 
of misappreciating the way in which the machine would 
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actually be used. The remark above to the effect that many 
tasks are 6600-size is a telling one. The machine is compa¬ 
rable in size to a 6600 and people want to use it that way. 
Big problems often imply big data and we failed to appreciate 
that during the initial design. 


The partitionable system 

When we first considered the possibility of building a 
multiprocessor in 1971, the ability to partition it into several 
disjoint subsystems was on our list of advantages for such 
architectures. While we are able to partition processors and 
memory, we are not able to run Hydra in more than one 
partition. 

C.mmp can be partitioned in such a way that some pro¬ 
cessors and memories can undergo maintenance and run 
stand-alone diagnostics without interfering with the larger 
partition running the operating system. 

The primary obstacle to running the operating system in 
two partitions is the money required to provide each par¬ 
tition with an adequate complement of I/O devices and 
memory. 

We do not know how to provide meaningful communi¬ 
cation between the capability structures of the two oper¬ 
ating systems. 

The principal effect of the failure to meet this goal has 
been that we must allocate disjoint time for users, hardware 
maintenance, and operating system testing. At present 28 
hours each week are reserved for maintenance. This parti¬ 
tioning has been very inconvenient for all concerned, and 
has certainly impeded progress on several occasions. Yet it 
seems clear that we have been unwilling to spend the money 
necessary to solve the problem—thus it seems safe to con¬ 
clude that the inconvenience has not been debilitating. 


(The lack of) human engineering 

As we have mentioned in several contexts previously, the 
human interface to the C.mmp/Hydra system is not well 
designed. To some extent this resulted from the novelty of 
the underlying system structure (we couldn’t anticipate 
some of the kinds of facilities that would be needed by users 
of either a capability-based or a multiprocessor system). To 
a large extent, however, the failure seems to have been one 
of having concentrated on the new, innovative aspects of 
the system and ignoring more mundane issues. 

There is a lack of human engineering in the operating 
system software which interacts directly with a user sitting 
at a terminal. 

It is difficult to pick up the minimal knowledge needed to 
know how to do useful things at a terminal. 


New users tend to have bad first impressions of the sys¬ 
tem. 

We did not realize how much work was required to make 
a smooth user interface and so did not allocate enough 
resources for it. 

We suspect the user environment would have received 
more work had the kernel implementors had to use it 
during their software development. (All kernel develop¬ 
ment and maintenance has been done on the PDP-10 com¬ 
puter, which has the Bliss-11 compiler and a linker for 
C.mmp.) 

One particular aspect of the human interface is especially 
interesting—the command language. It seems to be an al¬ 
most universal phenomenon that people don’t like whatever 
command language they have used in the past. We were no 
exception. Thus, rather than modeling our command lan¬ 
guage on any existing one, we chose to strike out in another 
direction. In particular, we chose to make the command 
language a (modest) interactive programming language— 
with declarations of variables, assignments, conditional and 
looping control constructs, macros, and so on. The power 
of this approach seems unquestionable, as is reflected by 
the following remarks. The remarkable thing (to the editors) 
is the lack of negative remarks during the meeting; the com¬ 
mand language usually comes under heavy attack on other 
occasions. 

The Command Language is much more flexible and pow¬ 
erful than the command scanners found on most systems. 

The concept of the Command Language as a programming 
language was good. 

The Command Language user on C.mmp is unique in 
having complete access to the Hydra environment. Sub¬ 
systems can almost be implemented directly in the Com¬ 
mand Language. 

Error reporting by the Command Language is poor. 

Another aspect of the human interface is the (lack of a) 
spectrum of programming languages: 

C.mmp lacks the wide range of languages available on 
conventional systems. 

The L* system provides its users with a complete envi¬ 
ronment compatible with that provided on the PDP-10 by 
its version of L*. 

The L* environment does not seem conducive to the con¬ 
struction of subsystems. 

The Algol 68 implementation on C.mmp gives users access 
to the multiprocess capabilities of C.mmp, but does not 
yet provide access to capabilities or the Hydra protection 
environment. 

The fact that most subsystem development takes place 
partially on C.mmp and partially on the PDP-lO’s (which 
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have Bliss-11 compilers) is not a severe hindrance now 
that smooth communication facilities exist between the 
machines. 

It is interesting (to the editors) that the word “baroque” 
was not used during the meeting; in other contexts it often 
is. Several features of Hydra and its subsystems do exhibit 
“second-system-itis”. There are things which are more gen¬ 
eral, and more complicated, than necessary. 


Project management 

The C.mmp/Hydra project was not a large project by most 
standards; there were never more than about 15 people, 
mostly students, working on the project at any one time. 
Nevertheless we made a number of errors which can only 
be classified as failures in the management of the project; 
taken together, these errors constitute one of our largest 
failures. 

Among our errors is a classic! Because the hardware and 
Hydra structures were new and exciting, we tended to focus 
on them to the exclusion of the more mundane things which 
also determine the ultimate utility of any system. This point 
recurred in many of the points raised at the meeting: 

The manpower allocated to the Policy Module was inad¬ 
equate. In fact this was true of all software outside the 
kernel. 

The failure to stress reliability and performance in the first 
PM was a mistake. 

The user environment was ignored at first because of our 
natural preoccupation with the Hydra kernel and the re¬ 
search problems it embodied. 

We underestimated how much work would be involved in 
constructing the user environment. 

We have a much better idea now about the proper struc¬ 
ture (or at least an adequate one) of the user environment 
than we did when we began building the first subsystems. 
Implementing basic concepts such as “jobs” and “termi¬ 
nals” in nonpriviledged software has subtle design and re¬ 
liability implications which we are just now appreciating. 

The management style used throughout the project was 
informal. There were very few memos, formal design re¬ 
views, or the other mechanisms of tight management con¬ 
trol. In most ways this felt appropriate to the academic 
environment and the high caliber of the individuals involved. 
It led to a number of problems, however, and the consensus 
of the meeting was that the management had been too loose. 
This is especially evident in the comments relating to a lack 
of formal specifications and the lack of uniform documen¬ 
tation and coding standards. 

The fact that the Hydra implementors did not have to use 
C.mmp for software development contributed to the ne¬ 
glect of the user environment. 


The lack of detailed hardware specifications hindered the 
parallel development of hardware and software but not 
the end result. 

Software was occasionally developed which took advan¬ 
tage of unspecified “features” of the hardware, making 
them difficult to change. 

Loose management coupled with the modularization tech¬ 
nique worked well except in forcing standardization of 
coding styles. 

We should not have depended on graduate students for 
complete software development for so long. Graduate stu¬ 
dents cannot keep deadlines reliably and are not tied to 
the project. [Furthermore, we feel that Ph.D. students 
should not spend an inordinate amount of time doing the 
standard programming chores which characterize any at¬ 
tempt to bring up a complete operating system.] 

Another class of management errors relates to what might 
be termed “public relations.” Being academics, we instinc¬ 
tively react somewhat negatively to the “attention-getting” 
aspect of PR, forgetting that its “information-providing” 
function is absolutely necessary. In a number of ways we 
failed to make information available publicly. 

Our problem is basically public relations—performance 
measurements indicate we have a winner on our hands. 

The lack of a smooth user environment was a deterrent 
to new users which could form the foundation of a happy 
and vocal user community. 

Since Hydra was not easily accessible to people outside 
the department, we could not adopt a “try it and see” 
attitude. 

Documentation is needed to encourage use internally and 
generate credibility externally. 

A DATA SAMPLER 

The previous section concludes our report of the meeting. 
Since the body of the report contains many subjective and 
unsubstantiated comments, we decided to include a few 
examples of the kinds of data on which these comments are 
based. We have chosen two examples: (1) a study of the 
effect of the small address problem on a specific user pro¬ 
gram, and (2) a study of the contention for locks in the 
Hydra kernel. 

A study of the small address problem 

The program used in this study of the SAP is HARPY. 
HARPY is a speech-understanding system which has been 
implemented on all of the departments major computers: 
C.mmp, a stand-alone PDP-11 running under UNIX, and the 
PDP-10 (both KA10, circa 1967, and KL10, circa 1976, pro¬ 
cessors are available in the department). Since HARPY ex- 
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Figure 1—A look at the small address problem 
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ists on all these machines, it makes a convenient benchmark. 
(We should point out that HARPY is not necessarily the 
best application for C.mmp, nor are the HARPY implemen¬ 
tations on C.mmp known to be optimal.) 

Figure 1 summarizes the data obtained from a series of 
experiments with HARPY working on a rather small task, 
namely a voice-input desk calculator that has a 37 word 
vocabulary. 

The horizontal dashed lines represent the performance of 
single-process implementations of HARPY on the depart¬ 
ments uniprocessors. The solid curves represent the per¬ 
formance of two implementations on C.mmp, both of which 
can utilize any number of processes. 

The two HARPY versions on C.mmp differ in their as¬ 
sumptions about the addressability of data. The “static map¬ 
ping” version knows that all of its data is always address¬ 
able, while the “dynamic mapping” version expects to have 
to do some mapping of relocation registers in order to ad¬ 
dress the data. In this second version, it must be realized 
that, in fact, all the data is addressable, and thus no operat¬ 
ing system overhead is involved. (The overhead is HARPY 
checking to see if relocation is necessary—it never is.) 

This type of data dramatically illustrates the effect of the 
SAP on performance—it costs nearly a factor of three in 
this example. The effect on programming difficulty is at least 
as great, but is not so easy to illustrate. 

Note that the one-process, static mapping version of 
HARPY runs very nearly as fast as the version running 
under UNIX, even though the C.mmp version has all the 
necessary mechanisms for multiprocessing. We think this 
indicates that the synchronization primitives (spinlocks in 
shared memory) do not contribute much overhead in this 
application. 

Also note that little improvement in performance is seen 
beyond three or four processes. This is simply due to a lack 
of work to do—the small vocabulary simply isn’t compli¬ 
cated enough to keep the processors busy. On larger vocab¬ 
ularies we typically see noticeable improvement out to 
eight processes. The upturn in the curves towards the end 
is due to the fact that all the faster PDP-11/40 processors 
are in use. As soon as one PDP-11/20 is used, the whole 
assemblage of processes slows down. This is because the 
particular decomposition of the algorithm limits the speed 
to that of the slowest process. 


A study of kernel lock contention 

One of the largest potential bottlenecks in a distributed 
operating system is contention for locks on shared data 
structures. The hardware monitor has been used to study 
this; the types of results obtained are shown in Figure 2. 

In this study, three programs with seemingly different 
demands on the system were run while the hardware monitor 
measured the activity on one processor. The data is illustra¬ 
tive only, since no claim is made that the programs in any 
way represented a “typical” system load. 

The principle result is that it seems we spend consistently 
less than 1 percent of the time blocked on locks. We do not 


Static 

1 

Program 

2 

3 

Total time of measurement 
(millis) 

17393 

32924 

20255 

Number of different locks 
detected 

53 

79 

181 

Average time inside a critical 
section (micros) 

279 

378 

279 

Total number of lock 
operations 

2955 

504 

4360 

Percent of locks which 
blocked 

5.5 

11.7 

6.1 

Percent of time spent in 
kernel code 

61.8 

16.9 

37.7 

Percent of time spent in 
blocked state 

.29 

.83 

.74 


Figure 2—A study of kernel lock contention 


yet have any measurement of the time lost due to blocking 
on semaphores. 


CONCLUSIONS 

The C.mmp/Hydra project has reached the point at which 
many of its most interesting and important results will 
emerge. With a growing user community, increasing relia¬ 
bility and a smoother user interface, we are in a position to 
gether data on various aspects of system performance under 
real loads. This data will augment that already collected on 
isolated algorithms to provide a comprehensive picture of 
C.mmp/Hydra performance. Along the way to constructing 
the current system we managed, in our opinion, to do some 
things well and some things not so well. This paper has been 
our attempt to report those opinions in the hope that others 
may benefit from our experiences. 
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INTRODUCTION 

A user oriented computer system is a computer system 
specifically designed to meet the users’ requirements. With 
the advances in solid state technology and the emergence of 
microprocessors, system designers are able to design spe¬ 
cialized systems to satisfy the users’ requirements. This 
trend has led to an era of user oriented design. However, 
current approaches to the design of computer systems and 
their evaluation, unfortunately, are based primarily on ex¬ 
perience and intuition. The specification, design, implemen¬ 
tation, and evaluation of large embedded computer systems, 
such as the air traffic control systems, the patient monitoring 
systems and the ballistic missile defense systems (BMD), 
are very expensive, difficult to test adequately, slow to 
deploy, and difficult to adapt to changing requirements. 1 
The major cause of these problems is the largeness of the 
systems. The activities of the systems are so varied and so 
complex that they are beyond the grasp of a single individ¬ 
ual. For example, the BMD systems include, besides the 
data processing subsystem, the radar and missile subsys¬ 
tems. Each of these subsystems requires special expertise 
to design, implement, and enhance its operations. As a re¬ 
sult, each subsystem is usually developed and maintained 
by a group of experts who have little knowledge of the other 
subsystems. This produces great difficulties in synchroniz¬ 
ing and optimizing the development process. One common 
problem has been that some final decisions on primitives 
(essential system characteristics) are made in one subsys¬ 
tem, generally without considering the overall system re¬ 
quirements. These early commitments bias the development 
process and force and restrict the choices of the other prim¬ 
itives to accommodate them, which, in turn, impose undue 
constraints on design freedom and reduce the flexibility dur¬ 
ing integrating and interfacing the system. As a result, the 
development process is more expensive and time consuming 
than it should be, and the design that is obtained is usually 
far from optimum. 

Another problem faced in the development of large com¬ 
puter systems is the ever changing system environment. 
When the system application changes or the technology 
changes, the system has to be modified to adapt to the 
changes. However, more too often, systems are designed 


without taking into account the provision for future changes. 
When the system evolves, the changes are incorporated into 
the system in a very disorganized manner. As a result, the 
unstructureness (or the entropy) of the system increases 
enormously 2 and leads to a regenerative, highly non-linear, 
increase in the effort and cost of system maintenance. 3 In 
addition to this, the reliability and the integrity of the system 
are also jeopardized greatly. 

In large scale critical real time systems, such as the nu¬ 
clear reactor control systems, the development process is 
further complicated by the real time constraints and the 
criticalities of the systems. These systems must perform all 
the required functions correctly within the given time limits, 
otherwise a large penalty has to be paid (for example, large 
loss of life due to a nuclear explosion). However, these 
systems cannot be tested in its real operational environ¬ 
ments. As a result, system validation has to rely heavily on 
analysis and simulation. Due to the high complexity of the 
systems, exhaustive testing will be impossible. The only 
solution is to design the system in an orderly way so that 
both validation and testing can be done efficiently. 

In this paper, a systematic design and development meth¬ 
odology for user-oriented data processing systems (DPS’s) 
is presented. The methodology will provide guidelines for 
the systematic design and construction of DPS’s so that the 
users’ requirements are satisfied if there exists a feasible 
design under the given constraints. However, it must be 
emphasized that it does not mean the whole design process 
can be automated. The engineering decisions will often be 
very complex and dependent on the experience of the de¬ 
signers. The design methodology will provide design laws 
and analysis tools to help the designers in making design 
decisions, trade-offs and predicting the consequences. By 
following the guidelines given by the design methodology, 
the very complex design process will be simplified and we 
will be able to develop reliable, effective, modifiable systems 
with low costs and lead times. 


Characteristics of the methodology 

The philosophy behind the methodology is based on hi¬ 
erarchical modelling of a DPS. The objective of establishing 
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the hierarchy is to map the overall system requirements 
successively into lower levels of finer detail. At the top level 
in the hierarchy, the requirements described are abstract 
and the coupling between various attributes and associated 
functions may be loose. As we proceed down the levels, the 
characteristics of various functions and the attributes are 
elaborated and become more specific, Figure 1. The use of 
abstraction at the top level allows a designer to initially 
express the system requirements in a very general manner 
and with little regard to the details of the design and imple¬ 
mentation. These initial system requirements are then re¬ 
fined in a step by step manner by gradually introducing more 
and more details (e.g., constraints and attributes) of the 
system. This combination of abstraction and stepwise re¬ 
finement enables the designer to overcome the problem of 
complexity inherent in the construction of a complex DPS 
by allowing him to concentrate on the relevant aspects of 
his design incrementally at any given time, without worrying 
about other details. By this hierarchical approach, the as¬ 
sumptions and decisions made throughout the design proc¬ 
ess can be traced systematically and any revisions or mod¬ 
ifications of the design as a result of the design development 
can easily be incorporated. In summary, the design meth¬ 
odology will: 

(1) guarantee that the architecture (statement of need, 
system objectives, and constraints) of the problem will 
be preserved. 

(2) support orderly evolution of the system satisfying the 
constraints such as performance, reliability, etc. with¬ 
out major revisions. 

(3) provide formal (mathematically rigorous) basis for the 
approach allowing precise evaluation of completeness, 
consistency and correctness of requirements at any 
level of definition. 

(4) represent effectively and efficiently the decision mak¬ 
ing constraints of a DPS by a specification language. 

(5) provide design attributes and documentation for evo¬ 
lution (growth and modification) so that changes can 
be made without reconsidering the whole design proc¬ 
ess. 


Overview of the methodology 

The design methodology proposed here can be broken 
down into four successive phases, Figure 2: 

(1) requirement and specification phase 


(2) design phase 

(3) implementation phase 

(4) evaluation and validation phase 

The requirement and specification phase starts with some 
(possibly incomplete, vague, and informal) users’ require- 



DEVELOPMENT 

COMPLETE 

Figure 2—Proposed design methodology 
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ments that approximate the desired system, and finishes 
when the modified and elaborated requirements have been 
formally encoded and tested to the satisfaction of the system 
engineers and the “customers”. This is the most difficult 
but also the most important step in the development process. 
Experience has shown that many design failures are due to 
either ill-defined (inconsistent and unclear) requirements or 
misinterpretation of the original problem statement. These 
account for up to 85 percent of requirement errors. 4,5 In 
order to avoid the above mistakes, the concept of a closed 
system is used. The users and their environment are consid¬ 
ered together during the requirement and specification proc¬ 
ess. Requirement elaboration is done jointly in the environ¬ 
ment and system models. By doing this, more complete and 
consistent requirements can be produced. 

The design phase starts'with the requirement specifica¬ 
tions and finishes when the system specifications are pro¬ 
duced. The objective is to organize and optimize the system 
in a well formed structure. It involves a hierarchy of de¬ 
compositions and partitioning of the system into subsys¬ 
tems. Decomposition is the process of dividing the system 
into several levels of components and subcomponents, and 
partitioning is the process of grouping these components and 
subcomponents into subsystems so to minimize the amount 
of interactions and to satisfy the performance and reliability 
constraints of the system. After each decomposition and 
partition step, the subsystems are verified to be consistent 
to the original system. Any discrepencies and mismatches 
are corrected before they can propagate into the next level. 
After the design process, the system functions will be well 
specified and will be ready for implementation. 

The implementation phase takes the system specification 
and develops the system architecture. It then maps the sys¬ 
tem functions into either hardware or software functions. It 
is only at this step that physical constraints and technology 
come into consideration. 

The final step is the evaluation and validation of the sys¬ 
tem. This phase uses the bottom up validation approach. It 
takes the final design and ensures that the system meets the 
original requirements. This step uses both analytical mod¬ 
elling and simulation. Mistakes or unfulfilled requirements 
are traced back to the source of the error. The system is 
then redesigned from that point. Since the system is broken 
down hierarchically, only the subsystems affected by the 
error and therefore only those that are stemming from the 
error point have to be redesigned. 

It should be mentioned that the development process is 
not a straight top down process. Tests and checks are con¬ 
ducted throughout the development process. Whenever er¬ 
rors are found, the design is backed up to the previous level. 
Therefore there is a feed-back path from each development 
process back to the previous one as indicated in Figure 2. 

DESIGN METHODOLOGY 

The design methodology is aimed at satisfying a broad 
spectrum of data processing applications including real time 
applications. Its primary objective is to develop reliable, 


effective, modifiable systems with low cost and lead time. 
In the following sections, the requirement and specification 
phase and the design phase of the methodology will be 
discussed in detail. The implementation and validation 
phases are too technology and architecture dependent and 
are beyond the scope of this paper. 

REQUIREMENT AND SPECIFICATION PHASE 

The requirement and specification phase starts with in¬ 
formally specified users’ needs and elaborates on them to 
generate the formal system requirement specifications. 
These specifications are used for two purposes: (1) as a 
problem definition for the design process, (2) as a means 
against which an implementation can be validated. The suc¬ 
cess in the development of the data processing system 
greatly depends on the correct interpretation of the require¬ 
ments in the specification phase. However, these require¬ 
ment specifications usually suffer from many ills: they are 
often designs rather than a statement of need; they are 
usually incomplete and are expressed in an ambiguous lan¬ 
guage (English); they are difficult to verify and hence are 
often incorrect, conflictive within themselves (inconsisten¬ 
cies); they are difficult to test and if one wants to modify 
them, it is difficult to locate and accurately modify all af¬ 
fected areas. In order to rectify the above problems, a sys¬ 
tematic procedure is used to generate correct, consistent, 
complete, traceable, design free and feasible requirements. 
Formal definitions of the above terms can be found in Ref¬ 
erence 6. 

In this methodology, the requirement and specification 
phase consists of four major steps, Figure 3: (i) requirement 
elaboration, (ii) requirement specification and attribute for¬ 
mulation, (iii) process definition, and (iv) verification of re¬ 
quirements. 



PHASE 

Figure 3—Requirement and specification phase 
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Figure 4—Functional decomposition of BMD management system 


Requirement elaboration 

The requirement elaboration step can be considered as a 
problem understanding stage. The requirement engineers 
investigate in great detail the users’ needs and develop clear 
and precise requirements of the system. In this step, the 
closed system approach is chosen. In a closed system, the 
designed system and its external environment are considered 
as a single entity. All activities in the closed system are 
investigated in a homogeneous way. After the requirement 
engineers have fully understood the operations of the closed 
system, the data processing requirements and the attributes 
of the computer system can be derived from the behavior of 
the computer system. These overall system requirements 
can then be decomposed into finer detail according to their 
functional, logical or precedence relationships. For example, 
the BMD management system can be decomposed function¬ 
ally as shown in Figure 4. Test experiments can also be 
generated from the behavior of the environment to verify 
the proper operation of the designed system. In this way, 
more complete and consistent requirements can be gener¬ 
ated. 

Requirement specification and attribute formulation 

From the required behavior of the system, the users’ 
original objectives can be formally expressed in the system 
requirements and system attributes. 1 The system require¬ 
ments are the objectives and constraints which the system 
must satisfy. Any system which meets the requirements is 
a candidate solution to the users’ problem. Attributes, on 
the other hand, specify either options or evaluation criteria 
for qualitative comparisons of competing systems that meet 
the system requirements. They are used to specify the pref¬ 
erences of the users. The generation of these system re¬ 
quirements and attributes is the requirement specification 
and attribute formulation step. As pointed out previously, 
one of the greatest problems in requirement specifications 
is the misinterpretation of the original system requirements. 
A plausible solution is to use dual specification teams to 
develop the system specifications from the requirements 
independently as in the development of critical real-time 
software for nuclear power plants. 7 The two specifications 
are then compared and discrepancies are resolved to the 
satisfaction of both teams. By this dual specification ap¬ 
proach, most of the errors due to ambiguities and misinter¬ 


pretations can be corrected before they can propagate into 
the next phase. 


Requirement specification 

The system requirements can be broken down into two 
categories: (i) functional requirements, and (ii) performance 
requirements. 

Functional requirements specify the input (stimulus) and 
output (response) relations of the system. These input to 
output mappings can be expressed vigorously in mathemat¬ 
ical formulas 8 or less formally in a specification language. 9,10 
Mathematical formulation defines explicitly the ranges of 
input and output domains and exhibits the required system 
functions by tabulation or a set of mathematical expressions. 
This allows formal consistency and correctness proofs of 
the system. However, this method is greatly limited by the 
ability of the requirement engineers in formulating the math¬ 
ematical functions. In real-world situations, the problems 
are usually so complex that pure mathematical formulation 
is impossible. Therefore in this methodology, the approach 
of using a specification language is chosen. 

A specification language is a syntactically and semanti¬ 
cally well defined language possibly intermixed with math¬ 
ematical equations. Its whole purpose is to provide an effi¬ 
cient and effective medium for defining the functional 
requirements. Several specification languages have been de¬ 
veloped previously. 5,9-12 This paper does not intend to de¬ 
velop a new specification language. We will choose a 
specification language and express the functional require¬ 
ments in it. In choosing the specification language, the con¬ 
structability and comprehensibility of the language must be 
evaluated carefully. The specification language must be able 
to express the functional requirements efficiently and be 
easily understood by the customers and the requirement 
engineers. It must be able to specify the system require¬ 
ments unambiguously and provide capabilities of perform¬ 
ance specifications in the case of real time system. The 
language should be amenable to both static (hierarchical 
relationship, data definition, etc.) and dynamic (control flow 
and data flow) analyses. Finally, it should be backed up by 
a specification data base management system and powerful 
graphical supports to provide easy and efficient accesses to 
the designed system. 

Performance requirements specify the functional effec¬ 
tiveness of the system. They include the input and output 
rates, response time, accuracy etc. There is essentially no 
notion of completeness as far as performance requirements 
are concerned. The requirement engineers have to work 
closely with the users to ensure that all the important per¬ 
formance aspects of the system are captured in the specifi¬ 
cations. At this point, it can be noted that there are still 
some uncertainties and vagueness in the methodology. How¬ 
ever, these are unavoidable as design in general is a wicked 
problem. 13 There is no stopping rule and definite test to the 
solution of a wicked problem. One stops only because one 
has run out of resources, patience, etc. Therefore, the best 
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Figure 5—Reliability payoff tree 


we can do is try to provide guidelines in the development 
process to improve the quality of the product. 

Attribute formulation 

Attributes represent the customers’ preference on the de¬ 
signed system, e.g., cost, reliability, availability, flexibility, 
expandability, reconfigurability etc. They are the evaluation 
criteria used to determine which characteristics make one 
system desirable over another system even though they may 
both meet the requirements. Usually, the system attributes 
are interdependent on each other and they may compete and 
interact with each other. In order to clearly specify the 
preferences of the customers, the system attributes are 
weighted or ranked according to the customers’ preferences. 
They are then expressed in a utility function 14 * 15 or in a pay¬ 
off tree. 16 Utility function has been used extensively in eco¬ 
nomics and in business management. It measures formally 
the degree of satisfaction of the customers in terms of the 


utility of the system. Once the utility function of a system 
has been formulated, the optimal system configuration can 
be determined by optimizing its utility. However, due to the 
inherent complexity of a large computer system and the high 
interdependency of its attributes, the formulation of its util¬ 
ity function can be very difficult. In such cases, the less 
formal approach of using payoff tree is preferred. It involves 
first generating the payoff trees of the system attributes. For 
example, the reliability payoff tree is shown in Figure 5. The 
payoff measure interrelationship can then be generated, Fig¬ 
ure 6. These payoff trees show clearly to the designers the 
tradeoffs involved in the attributes. Based on these payoff 
trees, trade-off decisions in later stages of the development 
process can be facilitated. This approach is being used in 
the development of the BMD system. 

Process definition 

The process definition step accepts inputs from the re¬ 
quirement specification and attribute formulation step and 
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identifies major functions to be performed. First the input 
stimulus and the required responses are characterized. This 
involves stating the form of the input and output signals. 
They may be mechanical, electrical, optical, etc. From this, 
the required I/O processes can be defined. For example, in 
an air traffic control system, it contains the radar control 
process, the graphic display process and the interactive I/O 
process. 

In parallel to the formulation of the I/O processes, the 
functional requirements can be decomposed into data proc¬ 
essing requirements, communication requirements, prece¬ 
dence constraints, etc. Similarly, the performance require¬ 
ments can be decomposed into resource requirements, 
scheduling requirements, etc. Based on these requirements 
and the attributes defined previously, the information flow 
and control flow of the system can be modelled and analyzed 
to identify the major operations to be performed and their 
locations of occurrence. From these analyses, the system 
processes required to perform the above functions can be 
defined. These process definitions state precisely the func¬ 
tion of the processes, the resources required by the pro¬ 
cesses, and the interaction between the processes. At this 
stage, the virtual system is formed and is ready for verifi¬ 
cation. For example, in the formulation of a tangent func¬ 
tion, it can be decomposed functionally into a sine function, 


a cosine function and a division operation, Figure 7a. The 
attributes of the function (like accuracy, delay, output range, 
cost, etc.) can then be extended to become the attributes of 
the component functions, Figure 7b. In this manner, the 
virtual system is formed. 

Verification of requirements 

In this step, the processes of the virtual system is verified 
to meet the original users’ requirements. As the system is 
developed hierarchically, the specifications of one level are 
the requirements of the next level. To verify the correctness 
of the virtual system, we only have to verify the consistency 
between the specifications and requirements between con¬ 
secutive levels. This simplifies the verification process a lot 
and if an analysable specification language is used, the con¬ 
sistency can be verified automatically. In addition to this, 
test experiments generated automatically 17 or formulated 
from the environment in the requirement elaboration step 
can be used to check the quality of the virtual system. If the 
tests are not acceptable, necessary changes are incorporated 
into the requirement specifications. The affected processes 
will be updated and the corrected system will be tested 
again. 
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Figure 7a—Functional Decomposition 


DESIGN PHASE 

The design phase starts with the defined processes which 
are the output of the requirement and specification phase. 
The major steps involved in the design phase are process 
decomposition and partitioning, functional specification and 
finally verification, Figure 8. Basically, the design phase 
requires a provision to trace the system requirements 
through all levels of design, and a means of assessing trade¬ 
offs at the functional level and comparing design alterna¬ 
tives. 

In the process partitioning phase the designer will decom¬ 
pose the system into a network of interacting tasks. The 
motivation for such partitioning is to increase modularity 
and testability and to decrease the interferences between 
processes. 18 This partitioning is based on the philosophy of 
hierarchical decomposition of the system into progressively 
more detailed networks of processes at different levels of 
abstraction. The hierarchical approach permits partial or¬ 
dering and allows the designer to isolate interacting and non¬ 
interacting parts. The partitioning criteria and approaches 
depend on the level at which the partitioning is carried out 
and upon the nature of the logical aspects involved, for 
example functional decomposition, algorithmic decomposi¬ 
tion and attributes decomposition. Several techniques can 
be adopted to perform partitioning at various levels. Our 
approach to this problem of partitioning is discussed in the 
following subsections. 



(e^) (gq) ( c q) 


Figure 7b—Attribute decomposition 



IMPLEMENTATION 

PHASE 

Figure 8—Design phase 


Decomposition and partitioning 

After the requirement process, a well defined, complete, 
consistent, unambiguous and testable set of system process 
specifications are produced. These system process specifi¬ 
cations insure that the system behavior will be satisfactory 
to the customer and that the required system can plausibly 
be designed and implemented. Usually the specified system 
at this stage is so complex that it is very difficult if not 
impossible for the hardware designers to start implementing 
the system. In order for the design process to be managea¬ 
ble, it must be decomposed in such a way that most deci¬ 
sions can be made locally, based on data available within a 
local area of the developing system specifications. 

The decomposition process can be accomplished by iden¬ 
tifying the tightly coupled processes of the system and fac¬ 
toring the system into subsystems with respect to this cri¬ 
terion. The remaining steps of the development process can 
then independently (or nearly so) elaborate the design of 
each subsystem, maintaining the inter-subsystem interactions 
as design invariances. However, the decomposition must be 
carried out very carefully satisfying the following criteria: 

(1) The decomposed system must be well-defined—they 
are consistent, complete, unambiguous and testable. 

(2) The decomposed system must be capable of being 
integrated. 
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(3) The decomposed system must meet the resource al¬ 
location requirements, reliability requirements, per¬ 
formance requirements, etc. 

(4) The decomposed system must satisfy the parametric 
logical specifications such that minor changes in the 
requirements would not require a redesign of the 
whole system. 

(5) The decomposed system must be expandable so that 
future growth of the system can be easily incorpo¬ 
rated. 

In accomplishing the above criteria, a decomposition pro¬ 
cedure will be used to guide the designer to generate a 
satisfactory decomposition of the system. The objective here 
is not to produce an optimal solution, but to develop a set 
of tools to help him to make his decisions. Most of the steps 
in the decomposition process will be automated to relieve 
the designer from tedious computations. However, some 
tradeoff decisions require human experience and interaction, 
and must be made by the designer. As a result, a close 
interaction of the designer in the decomposition process is 
required. 


Preliminary decomposition 

A large system will have many processes that are only 
loosely coupled, for example, the communication process 
and the data processing process. These loose subsets of 
tightly coupled processes can easily be identified by the 
designer and can be used to partition the system into sub¬ 
systems. Thus, the whole system is decomposed into smaller 
subsystems, each representing some aspects of the original 
system from a different point of view. The number of such 
decomposed subsystems is application-dependent and sen¬ 
sitive to the identification of loosely coupled subsets of 
processes. By this preliminary decomposition, the system 
is decomposed into a structure according to its requirements 
and is ready for further analyses. 


Decomposition based on interaction 

The objective of interaction decomposition is to reduce 
the amount of interactions between subsystems. This in turn 
reduces the complexity of the interfaces and the amount of 
communication between subsystems. The interaction de¬ 
composition process can be divided into two cooperating 
steps: (i) decision process which requires direct interaction 
from the designer, and (ii) solution finding process which 
can be highly automated, Figure 9. In the decision process, 
the designer looks at the requirement specifications and 
assigns weights to the interaction between processes. These 
weights will be a function of communication cost, amount 
of traffic flow, degree of synchronization, ranking of the 
attributes, etc. By using this weighting function, tightly cou¬ 
pled processes will be assigned a high weight. Processes 
that should be in different modules (for example, due to 
reliability requirement) are assigned a very low weight. With 



Figure 9—Decomposition based on interaction 


this weight assigning function, we will then model the system 
by an interaction graph, Figure 10. In the interaction graph, 
nodes represent processes and the weight assigned to each 
arc corresponds to the amount of interaction of the two 
processes connected by the arc. In this way, the system can 
then be decomposed into subsystems automatically by the 
solution finding process. After the decomposition, the de¬ 
composed system is then checked by the designer to deter¬ 
mine whether all the requirements are satisfied. The dis¬ 
crepancies are identified and the required changes are 
imposed onto the interaction graph which is analyzed by the 
solution finding step again. These two steps are iterated until 
all the specifications are met. 

Solution finding procedure: In this step, the system which 
is represented by an interaction graph is partitioned into 
loosely coupled subsystems such that the system require¬ 
ments are satisfied. First, the interaction cut-tree will be 
generated from the interaction graph by using the max. flow 
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cut 




min. cut algorithm. 19 It involves choosing two nodes arbi¬ 
trarily in the interaction graph and finding the minimum cut 
set of the two nodes by the max. flow min. cut algorithm. 
This minimum cut divides the interaction graph into two 
subgraphs. Two other nodes are then chosen arbitrarily in 
one subgraph and the above procedure is repeated (with the 
other subgraphs being considered as macro nodes) until the 
interaction cut tree is generated. By this transformation, the 
interaction requirements between processes are represented 
very clearly by the interaction cut-tree, Figure 11. The min¬ 
imal cut-set separating two nodes in the interaction graph is 
one to one corresponding to the minimal cut-set of the same 
two nodes in the interaction cut-tree. In addition to this, the 
values of the corresponding cut-sets in the two graphs are 
equal. For example, the minimal cut-set separating nodes a 
and f partitions the interaction graph and the interaction cut- 
tree identically into two subgraphs, {a,b} and (c,d,e,f). The 
two cut sets both have values equal to 22. As a result, the 
decomposition can be performed on the interaction cut-tree 
rather than the interaction graph. This simplifies the com¬ 
putation greatly and displays clearly to the designer the 
condition of the system. The minimum cut can be identified 
easily in the cut-tree as it is the minimum weighted arc in 
the unique path connecting the two nodes. 

After the interaction cut-tree has been generated, the sys¬ 
tem can be partitioned into loosely coupled subsystems such 
that the total weight of all the arcs in the cut-set is minimum. 
In addition to this, the partition should preserve the special 
configuration imposed by the requirements specified by the 
designer. For example, because of resource requirements, 
processes a and f, process a and d, and processes e and f 
have to be in different modules. In order for a and d to be 
on different modules, either arc (a,b), (b,f) or (f,d) has to be 
cut in the cut-tree. These requirements are then expressed 
in a table as shown in Table I. 

The decomposition can now be achieved by finding the 
set of arcs in Table 1 such that each row in the table contains 


TABLE I.—Constraint Table 


arcs 

separating 

(a,b) <b,f) (c,f) (d,f) (e,f) 

a, f 

X 





a, d 

X 

X 

X 



e, f 







at least one cross in the arcs chosen. This can be solved by 
the set covering algorithm. 20 In our example, arcs (b,f) and 
(e,f) are chosen such that the interactions between subsys¬ 
tems are minimized, Figure 12. In general, this method will 
give a solution very close to the optimal solution and the 
computation complexity is very low when compared with 
that of generating the optimal solution. 


Functional specification 

The next major step in the design phase is functional 
specification of the partitioned processes. This functional 
specification is different from the process specification de¬ 
scribed in the requirement specification phase. The objective 
of the process specification is to define the interactions of 
the processes for the decomposition step. The objective of 
the functional specification here is to define the character¬ 
istics of the functions so as to enable optimization in the 
functions to processors mapping. 

In the functional specification, all the processes in the 
same partition are considered as a single function. Similar 
to the process specification, the input and output relation, 
the precedence constraints and the interactions between dif¬ 
ferent functions are determined. In addition to these, the 
characteristics of the function are defined. These include: 

(i) types of operations to be performed—matrix opera¬ 
tions, floating point or integer operations, etc. 

(ii) resource requirements—storage requirement, proc¬ 
essing power, etc. 

(iii) speed requirements—frequency and execution speed 
of the function. 

Based on these information and the processors available, 
the functions are mapped onto the available processors. In 
Reference 21, an efficient mapping algorithm of two proces¬ 
sors system are discussed. For the general case of n pro¬ 
cessors, no efficient algorithm is known at this time and 
more research should be done in this area. 


Verification of design 

In order to be able to verify the correctness of the design 
and to evaluate the effectiveness of the control, the system 
must be modelled in some abstract model. This abstract 
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model should be amenable to analyze the intercommunica¬ 
tion, the synchronization, the performance and the coordi¬ 
nation of the functions. In this methodology, the Petri net 
model is chosen. 

Petri nets display clearly the flow of information and con¬ 
trol in systems, especially those which exhibit asynchronous 
and concurrent properties. However, in order to model the 
time constraints of real-time systems, the Petri net model is 
extended to include the execution time aspect of a system. 
In the following sections, the extended Petri net model and 
the analysis techniques will be discussed. In general, the 
designed system is modelled by a Petri net at all levels in 
the design phase. The Petri net is analyzed to guarantee the 
proper behavior of the system. For example, the Petri net 
models of the system before and after the decomposition are 
analyzed to ensure the consistency of decomposition. 





Figure 14—Reachability graph 


Analyses of the system 

Throughout the whole design phase, the system is mod¬ 
elled by a Petri net. By analyzing the liveness, the bound¬ 
ness and the proper termination properties of the Petri 
net, 22-25 the properties of the designed system can be un¬ 
veiled. A Petri net is live if for each transition in the net, 
there always exists a firing sequence to fire it. By the live¬ 
ness property of the Petri net, the designed system is guar¬ 
anteed to be dead-lock free. A Petri net is bounded if for 
each place in the net, there exists an upper bound to the 
number of tokens that can be there simultaneously. By the 
boundness property of the Petri net, the number of buffers 
required between asynchronous processes can be deter¬ 
mined and therefore information loss due to buffer overflow 
can be avoided. A Petri net is properly terminated if the 
Petri net always terminates in a well-defined manner. By the 
proper termination property, the designed system is guar¬ 
anteed to function in a well behaved manner. (This point 
will be elaborated later.) 

All the above three properties can be analyzed by con¬ 
structing the reachability graph. (The method for construct¬ 
ing the reachability graph can be found in Reference 22.) In 



Figure 13, the Petri net models a concurrent system with 
five processes: r, s, t, u and v. The reachability graph is 
shown in Figure 14. Since the reachability graph is strongly 
connected and all the entries are finite, the concurrent sys¬ 
tem is live and bounded. By these two properties, the con¬ 
current system is guaranteed to be dead-lock free and to 
have enough buffers. 


Extended Petri net 

However in real world problems, especially in real-time 
systems, the execution time of a process is a very important 
aspect of a system. In order to model this aspect, the Petri 
net model is extended to include the notion of time. In the 
extended net, each transition is associated with two times, 
tl and t2, where 

tl=minimum time a transition can begin firing after 
being enabled (it can be zero) 

and /2=maximum time a transition can remain not fired 
after being enabled (t2>tl) 

Whenever a transition is enabled, it has to fire between 
times tl and t2. In the case that a transition has (tl,t2) as 
defined and an execution duration of u, it can be modelled 
by cascading two transitions with times (t\,t2) and ( u,u ) 
associated to the transitions, Figure 15. By using this ap- 
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Figure 15—Timed Petri net 
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proach, the problem of losing tokens during the firing of a 
transition is avoided. 

By using the extended Petri net, the concurrent system 
discussed previously is analyzed again. The Petri net model 
of the concurrent system and its time constraints are shown 
in Figure 16. A careful study of the system will show that 
transition u is dead (it will never be fired). This is due to the 
long delay produced by transition t. After firing transition 
r, the token in place C will move into place E and then move 
into place A before the token from place D can move into 
place F. This situation is shown in the reachability graph in 
Figure 17. The two times associated to each arc is the ear¬ 
liest and the latest transition times of the arc. The two arcs 
t 1 and t 2 are blocked because their earliest transition time is 
longer than the latest transition time of the other transition 
emitting from the same node. Therefore by using the ex¬ 
tended Petri net, the dead-lock situation due to time con¬ 
straints can be detected. In addition to that, if the Petri net 
is a marked graph 26 and if the execution time, t, of a tran¬ 
sition is taken to be t2, the maximum computation rate, p, 
of each of the transition can be computed by 26 

p=min |^r-1 k= 1,2,. . . «j 

where 

T k = I T, 

tieCk 



is the sum of the times associated with transitions of circuit 
C fe and 

N k = 2 M k 

V k eC k 

is the number of tokens in places of circuit C. 

In finding the dead-lock situation in the extended Petri 
net, the algorithm to find the earliest and latest transition 
times in the reachability graph is quite lengthy and compli¬ 
cated. For example, the arc t 2 is associated with times (3,5) 
rather than (5,6) because transition t is enabled before tran¬ 
sition s is fired. This type of interdependency can be re¬ 
solved by tracing back the reachability graph to find the 
time when the transition is enabled. The computation can 
be done by a computer. Although it may be very lengthy, 
it is worth the effort to guarantee the designed system to be 
dead-lock free. 


Proper termination 

A Petri net is properly terminated if a token is injected 
into the input place, the net produces a finite number of 
tokens and terminates with all the tokens in its output 
place. 23,24 This implies that if the corresponding system is 
initiated, it will always execute to completion without any 
intermediate results or pending conditions left in the system. 

The proper termination property of a Petri net can be 
studied by its reachability graph. For example, the Petri net 
and its reachability graph are shown in Figure 18 and Figure 
19. Since the only maximal node in the reachability graph 
is the state that place E contains a token, the Petri net is 
properly terminated. 

The notion of proper termination can be used to verify the 
consistency of decomposition. For example, in Figure 20, 
transition s in global net N1 is decomposed into the subnet 
S. Virtual nodes A through G are then added to the subnet 
to form N2 which is then analyzed for proper termination. 




Figure 17—Reachability graph 


Figure 18—A properly terminated Petri net 
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( 1 , 0 , 0 , 0 , 0 ) 



( 0 , 0 , 0 , 0 , 1 ) 


Figure 19—Reachability graph 

Global Net N1 


If N2 is properly terminated, the decomposition is consis¬ 
tent. This guarantees that no data stay in the decomposed 
process and no extra data are generated when the decom¬ 
posed process is substituted for the original process in the 
system. If time constraints, (fl,f2), are added to each of the 
transitions, the time constraints of transitions can be verified 
by finding the minimum and maximum path lengths in the 
decomposed net. 

The verification techniques discussed above are by no 
means complete. They are just some of the approaches for 
analyzing the synchronization and consistency of interacting 


N2 



Figure 20—Consistency verification by Petri net 
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asynchronous processes. It is hoped that through these anal¬ 
ysis techniques, synchronization errors in the designed sys¬ 
tem can be detected before implementation. 

IMPLEMENTATION, EVALUATION AND 

VALIDATION 

The implementation phase takes the virtual system and 
develops the system architecture. It then maps the system 
functions into either hardware or software functions. This 
step is greatly dependent on the technology, the architecture 
chosen, the physical constraints of the system. The final 
phase is the evaluation and validation of the system. This 
phase uses the bottom up validation approach. Both analyt¬ 
ical modelling and simulation will be used. Because of the 
hierarchical decomposition, each subsystem to be analyzed 
should be small and therefore the complexity should be low. 
Mistakes or unfulfilled requirements found are traced back 
to the source of error. The system is then redesigned from 
that point. 

CONCLUSION 

This paper has discussed a systematic design procedure for 
a user oriented computer system. The main objective is to 
develop reliable, effective, modifiable systems with low cost 
and lead time. The methodology uses the concept of abstrac¬ 
tion, stepwise refinement and modularity to design a DPS. 
By following the design guidelines of the methodology, the 
system can be developed systematically in a hierarchical 
manner. 

The design methodology is divided into four successive 
phases: (i) requirement and specification phase, (ii) design 
phase, (iii) implementation phase, and (iv) evaluation and 
validation phase. The first two phases have been explored 
in detail in the previous sections. In the requirement and 
specification phase, the requirements and the characteristics 
of the system are elaborated and expressed in a formal 
specification. The basic steps involved, the primitives to be 
specified, and the choice of specification language are dis¬ 
cussed. In the design phase, the system specifications are 
analyzed to generate the virtual system. A systematic de¬ 
composition procedure is proposed. Analysis and modelling 
techniques used in this phase are also discussed. By mod¬ 
elling the virtual system in an extended timed Petri net, the 
performance, the consistency and the liveness of the system 
can be determined. The last two phases of the methodology 
are only outlined briefly. They are too technology and ar¬ 
chitecture dependent and are beyond the scope of this paper. 
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INTRODUCTION 

Large virtual address space minicomputers 

Perhaps the most useful definition of a minicomputer sys¬ 
tem is based on price: depending on one’s perspective such 
systems are typically found in the $20K to $200K range. The 
twin forces of market pull—as customers build increasingly 
complex systems on minicomputers—and technology 
push—as the semiconductor industry provides increasingly 
lower cost logic and memory elements—have induced min¬ 
icomputer manufacturers to produce systems of considera¬ 
ble performance and memory capacity. Such systems are 
typified by the DEC PDP-11/70. From an architectural point 
of view, the characteristic which most distinguishes many 
of these systems from larger mainframe computers is the 
size of the virtual address space: the immediately available 
address space seen by an individual process. For many 
purposes the 65K byte virtual address space typically pro¬ 
vided on minicomputers (such as the PDP-11) has not been 
and probably will not continue to be a severe limitation. 
However, there are some applications whose programming 
is impractical in a 65K byte virtual address space, and per¬ 
haps most importantly, others whose programming is appre¬ 
ciably simplified by having a large virtual address space. 
Given the relative trends in hardware and software costs, 
the latter point alone will insure that large virtual address 
space minicomputers play an increasingly important role in 
minicomputer product offerings. 

In principle, there is no great challenge in designing a 
large virtual address minicomputer system. For example, 
many of the large mainframe computers could serve as ar¬ 
chitectural models for such a system. The real challenge lies 
in two areas: compatibility—very tangible and important; 
and simplicity—intangible but nonetheless important. 

The first area is preserving the customer’s and the com¬ 
puter manufacturer’s investment in existing systems. This 
investment exists at many levels: basic hardware (principally 
busses and peripherals); systems and applications software; 
files and data bases; and personnel familiar with the pro¬ 
gramming, use, and operation of the systems. For example, 
just recently a major computer manufacturer abandoned a 


major effort for new computer architectures in favor of 
evolving its current architectures. 1 

The second intangible area is the preservation of those 
attributes (other than price) which make minicomputer sys¬ 
tems attractive. These include approachability, understand- 
ability, and ease of use. Preservation of these attributes 
suggests that simply modelling an extended virtual address 
minicomputer after a large mainframe computer is not 
wholly appropriate. It also suggests that during architectural 
design, tradeoffs must be made between more than just 
performance, functionality, and cost. Performance or func¬ 
tionality features which are so complex that they appreciably 
compromise understanding or ease of use must be rejected 
as inappropriate for minicomputer systems. 


VAX-11 overview 

VAX-11 is the Virtual Address eXtention of PDP-11 ar¬ 
chitecture. 2,3 The most distinctive feature of VAX-11 is the 
extension of the virtual address from 16 bits as provided on 
the PDP-11 to 32 bits. With the 8-bit byte the basic address¬ 
able unit, the extension provides a virtual address space of 
about 4.3 gigabytes which, even given rapid improvement 
in memory technology, should be adequate far into the fu¬ 
ture. 

Since maximal PDP-11 compatibility was a strong goal, 
early VAX-11 design efforts focused on literally extending 
the PDP-11: preserving the existing instruction formats and 
instruction set and fitting the virtual address extension 
around them. The objective here was to permit, to the extent 
possible, the running of existing programs in the extended 
virtual address environment. While realizing this objective 
was possible (there were three distinct designs), it was felt 
that the extended architecture designs were overly compro¬ 
mised in the areas of efficiency, functionality, and program¬ 
ming ease. 

Consequently, it was decided to drop the constraint of the 
PDP-11 instruction format in designing the extended virtual 
address space or native mode of the VAX-11 architecture. 
However, in order to run existing PDP-11 programs, VAX- 
11 includes a PDP-11 compatibility mode. Compatibility 
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mode provides the basic PDP-11 instruction set less only 
privileged instructions (such as HALT) and floating point 
instructions (which are optional on most PDP-11 processors 
and not required by most PDP-11 software). 

In addition to compatibility mode, a number of other fea¬ 
tures to preserve PDP-11 investment have been provided in 
the VAX-11 architecture, the VAX-11 operating system 
VAX/VMS, and the VAX-11/780 implementation of the 
VAX-11 architecture. These features include: 

1. The equivalent native mode data types and formats are 
identical to those on the PDP-11. Also, while extended, 
the VAX-11 native mode instruction set and addressing 
modes are very close to those on the PDP-11. As a 
consequence VAX-11 native mode assembly language 
programming is quite similar to PDP-11 assembly lan¬ 
guage programming. 

2. The VAX-11/780 uses the same peripheral busses (Uni¬ 
bus and Massbus) as the PDP-11 and uses the same 
peripherals. 

3. The VAX/VMS operating system is an evolution of the 
PDP-11 RSX-11M and IAS operating systems, offers 
a similar although extended set of system services, and 
uses the same command languages. Additionally, 
VAX/VMS supports most of the RSX-11 M/IAS system 
service requests issued by programs executing in com¬ 
patibility mode. 

4. The VAX/VMS file system is the same as used on the 
RSX-11 M/IAS operating systems permitting inter¬ 
change of files and volumes. The file access methods 
as implemented by the RMS record manager are also 
the same. 

5. VAX-11 high level language compilers accept the same 
source languages as the equivalent PDP-11 compilers 
and execution of compiled programs gives the same 
results. 

The coverage of all these aspects of VAX-11 is well be¬ 
yond the scope of any single paper. The remainder of this 
paper discusses the design of the VAX-11 native mode ar¬ 
chitecture and gives an overview of the VAX-11/780 system. 

VAX-11 NATIVE ARCHITECTURE 
Processor state 

Like the PDP-11, VAX-11 is organized around a general 
register processor state. This organization was favored be¬ 
cause access to operands stored in general registers is fast 
(since the registers are internal to the processor and register 
accesses do not need to pass through a memory management 
mechanism) and because only a small number of bits in an 
instruction are needed to designate a register. Perhaps most 
importantly, the registers are used (as on the PDP-11) in 
conjunction with a large set of addressing modes which 
permit unusually flexible operand addressing methods. 

Some consideration was given to a pure stack based ar¬ 
chitecture. However it was rejected because real program 


data suggests the superiority of two or three operand instruc¬ 
tion formats. 4 Actually VAX-11 is quite stack oriented, and 
although it is not optimally encoded for the purpose, can easily 
be used as a pure stack architecture if desired. 

VAX-11 has 16 32-bit general registers (denoted R0-R15) 
which are used for both fixed and floating point operands. 
This is in contrast to the PDP-11 which has eight 16-bit 
general registers and six 64-bit floating point registers. The 
merged set of fixed and floating registers were preferred 
because it simplifies programming and permits a more ef¬ 
fective allocation of the registers. 

Four of the registers are assigned special meaning in the 
VAX-11 architecture: 

1. R15 is the program counter (PC) which contains the 
address of the next byte to be interpreted in the instruc¬ 
tion stream. 

2. R14 is the stack pointer (SP) which contains the ad¬ 
dress of the top of the processor defined stack used for 
procedure and interrupt linkage. 

3. R13 is the frame pointer (FP). The VAX-11 procedure 
calling convention builds a data structure on the stack 
called a stack frame. FP contains the address of this 
structure. 

4. R12 is the argument pointer (AP). The VAX-11 pro¬ 
cedure calling convention uses a data structure called 
an argument list. AP contains the address of this struc¬ 
ture. 

The remaining element of the user visible processor state 
(additional processor state seen mainly by privileged pro¬ 
cedures is discussed later) is the 16-bit processor status 
word (PSW). The PSW contains the N, Z, V, and C condi¬ 
tion codes which indicate respectively whether a previous 
instruction had a negative result, a zero result, a result which 
overflowed, or a result which produced a carry (or borrow). 
Also in the PSW are the IV, DV, and FU bits which enable 
processor trapping on integer overflow, decimal overflow, 
and floating underflow conditions respectively. (The trap¬ 
ping on conditions of floating overflow and divide by zero 
for any data type are always enabled.) 

Finally, the PSW contains the T bit which when set forces 
a trap at the end of each instruction. This trap is useful for 
program debugging and analysis purposes. 


Data types and formats 

The VAX-11 data types are a superset of the PDP-11 data 
types. Where the PDP-11 and VAX-11 have equivalent data 
types the formats (representation in memory) are identical. 
Data type and data format identity is one of the most com¬ 
pelling forms of compatibility. It permits free interchange of 
binary data between PDP-11 and VAX-11 programs. It fa¬ 
cilitates source level compatibility between equivalent PDP- 
11 and VAX-11 languages. It also greatly facilitates hard¬ 
ware implementation of and software support of the PDP-11 
compatibility mode in the VAX-11 architecture. 




VAX-11/780—A Virtual address extension to the DEC PDP-11 family 


969 


The VAX-11 data types divide into five classes: 

1. Integer data types are the 8-bit byte, the 16-bit word, 
the 32-bit long word, and the 64-bit quadword. Usually 
these data types are considered signed with negative 
values represented in two’s complement form. How¬ 
ever, for most purposes they can be interpreted as 
unsigned and the VAX-11 instruction set provides sup¬ 
port for this interpretation. 

2. Floating data types are the 32-bit floating and the 64- 
bit double floating. These data types are binary nor¬ 
malized, have an 8-bit signed exponent, and have a 25- 
or 57-bit signed fraction with the redundant most sig¬ 
nificant fraction bit not represented. 

3. The variable bit field data type is 0 to 32 bits located 
arbitrarily with respect to addressable byte boundaries. 
A bit field is specified by three operands: the address 
of a byte, the starting bit position P with respect to bit 
0 of that byte, and the size S of the field. The VAX-11 
instruction set provides for interpreting the field as 
signed or unsigned. 

4. The character string data type is 0 to 65535 contiguous 
bytes. It is specified by two operands: the length and 
starting address of the string. Although the data type 
is named “character string”, no special interpretation 
is placed on the values of the bytes in the character 
string. 

5. The decimal string data types are 0 to 31 digits. They 
are specified by two operands: a length (in digits) and 
a starting address. The primary data type is packed 
decimal with two digits stored in each byte except that 
the byte containing the least significant digit contains 
a single digit and the sign. Two ASCII character dec¬ 
imal types are supported: leading separate sign and 
trailing embedded sign. The leading separate type is a 
“ + “, or “(blank)” (equivalent to “+”) ASCII 
character followed by 0 to 31 ASCII decimal digit char¬ 
acters. A trailing embedded sign decimal string is 0 to 
31 bytes which are ASCII decimal digit characters ex¬ 
cept for the character containing least significant digit 
which is an arbitrary encoding of the digit and sign. 

All of the data types except field may be stored on arbi¬ 
trary byte boundaries—there are no alignment constraints. 
The field data type, of course, can start on an arbitrary bit 
boundary. 

Attributes of and symbolic representations for most of the 
data types are given in Table I and Figure 1. 


Instruction format and address modes 

Most architectures provide a small number of relatively 
fixed instruction formats. Two problems often result. First, 
not all operands of an instruction have the same specification 
generality. For example, one operand must come from mem¬ 
ory and another from a register; or one must come from the 
stack and another from memory. Second, only a limited 
number of operands can be accommodated: typically one or 


TABLE I.—Data Types 


DATA TYPE 

SIZE 

RANGE (decimal) 


Integer 

- 

Signed 

• Unsigned 

Byte 

8 bits 

-12810 +127 

Oto 255 

Word 

16 bits 

-32768 to+32767 

Oto 65535 

longword 

32 bits 

-2” to +2 3 '-1 

0 to 2 32 -1 

Quadword 

64 bits 

-2 ,3 to +2*’-1 

0to2**-1 

Floating Point 


±2.9 X 10- 3r to 1.7 X 10 3 * 


Floating 

32 bits 

approximately seven decimal 
digits precision 

Double Floating 

64 bits 

approximately sixteen decimal 
digits precision 

Packed Decimal 

String 

Oto 16 bytes 
(31 digits) 

numeric, two digits per byte 
sign in low half of last byte 

Character String 

0 to 65535 bytes 

one character per byte 

Variable-length 

Bit Field 

Oto 32 bits 

dependent on intrepretation 


two. For instructions which inherently require more oper¬ 
ands (such as field or string instructions), the additional 
operands are specified in ad hoc ways: small literal fields in 
instructions, specific registers or stack positions, or packed 
in fields of a single operand. Both these problems lead to 
increased programming complexity: they require superflu¬ 
ous move type instructions to get operands to places where 
they can be used and increase competition for potentially 
scarce resources such as registers. 

To avoid these problems two criteria were used in the 
design of the VAX-11 instruction format: (1) all instructions 
should have the “natural” number of operands and (2) all 
operands should have the same generality in specification. 
These criteria led to a highly variable instruction format. An 
instruction consists of a one or two* byte opcode followed 
by the specifications for n operands (n S: 0) where n is an 
implicit property of the opcode. An operand specification is 
one to 10 bytes in length and consists of a one or two byte 
operand specifier followed by (as required) zero to eight 


3* C 


OOUfllE FLOATING 


—~T*-®l 

FRACTION 


Sj_EXPONENT | FRACTION 

FRACTION 


. EXPONENT I 


PACKED DECIMAL STRING («1231 


FRACTION 

FRACTION 

FRACTION 

CHARACTER STRING (XVZ) 


Figure 1—Data formats 


* No currently defined instructions use two byte opcodes. 
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bytes of specifier extension. The operand specifier includes 
the address mode and designation of any registers needed 
to locate the operand. A specifier extension consists of a 
displacement, an address, or immediate data. 

The VAX-11 address modes are with one exception a 
superset of the PDP-11 address modes. The PDP-11 address 
mode autodecrement deferred was omitted from VAX-11 
because it was rarely used. 

Most operand specifiers are one byte long and contain 
two 4-bit fields: the high order field (bits 7:4) contains the 
address mode designator and the lower field (bits 3:0) con¬ 
tains a general register designator. The address modes in¬ 
clude: 

1. Register mode in which the designated register con¬ 
tains the operand. 

2. Register deferred mode in which the designated regis¬ 
ter contains the address of the operand. 

3. Autodecrement mode in which the contents of the des¬ 
ignated register are first decremented by the size (in 
bytes) of the operand and then used as the address of 
the operand. 

4. Autoincrement mode in which the contents of the des¬ 
ignated register are first used as the address of the 
operand and are then incremented by the size of the 
operand. Note that if the designated register is PC, the 
operand is located in the instruction stream. This use 
of autoincrement mode is called immediate mode. In 
immediate mode the one to eight bytes of data are the 
specifier extension. 

Autoincrement mode can be used sequentially to 
process a vector in one direction and autodecrement 
mode used to process a vector in the opposite direc¬ 
tion. Autoincrement, register deferred, and autodecre¬ 
ment modes can be applied to a single register to im¬ 
plement a stack data structure: autodecrement to 
“push”, autoincrement to “pop”, and register de¬ 
ferred to access the top of the stack. 

5. Autoincrement deferred mode in which the contents of 
the designated register are used as the address of a 
longword in memory which contains the address of the 
operand. After this use, the contents of the register are 
incremented by four (the size in bytes of the longword 
address). Note that if PC is the designated register, the 
absolute address of the operand is located in the in¬ 
struction stream. This use of autoincrement deferred 
mode is termed absolute mode. In absolute mode the 
4-byte address is the specifier extension. 

6. Displacement mode in which a displacement is added 
to the contents of the designated register to form the 
operand address. There are three displacement modes 
depending on whether a signed byte, word, or long¬ 
word displacement is the specifier extension. These 
modes are termed byte, word, and longword displace¬ 
ment respectively. Note that if PC is the designated 
register, the operand is located relative to PC. For this 
use the modes are termed byte, word, and longword 
relative mode respectively. 


7. Displacement deferred mode in which a displacement 
is added to the designated register to form the address 
of a longword containing the address of the operand. 
There are three displacement deferred modes depend¬ 
ing on whether a signed byte, word, or longword dis¬ 
placement is the specifier extension. These modes are 
termed byte, word, and longword displacement re¬ 
spectively. Note that if PC is the designated register, 
the operand address is located relative to PC. For this 
use the modes are termed byte, word, and longword 
relative deferred mode respectively. 

8. Literal mode in which the operand specifier itself con¬ 
tains a 6-bit literal which is the operand. For integer 
data types the literal encodes the values 0-63; for float¬ 
ing data types the literal includes three exponent and 
three fraction bits to give 64 common values. 

9. Index mode which is not really a mode but rather a one 
byte prefix operator for any other mode which evalu¬ 
ates to a memory address (i.e., all modes except reg¬ 
ister and literal). The index mode prefix is cascaded 
with the operand specifier for that mode (called the 
base operand specifier) to form an aggregate two byte 
operand specifier. The base operand specifier is used 
in the normal way to evaluate a base address. A copy 
of the contents of the register designated in the index 
prefix is multiplied by the size (in bytes) of the operand 
and added to the base address. The sum is the final 
operand address. There are three advantages to the 
VAX-11 form of indexing: (a) the index is scaled by 
the data size and thus the index register maintains a 
logical rather than a byte offset into an indexed data 
structure, (b) indexing can be applied to any of the 
address modes which generate memory addresses and 
this results in a comprehensive set of indexed address¬ 
ing methods, and (c) the space required to specify 
indexing and the index register is paid only when in¬ 
dexing is used. 

The VAX-11 assembler syntax for the address modes is 
given in Figure 2. The bracketed ({ }) notation is optional 


Literal 

(Immediate) 


Register 

R/i 

Register Deterred 

<R n) 


Autodecrernont 

-(R/i) 


Autoincrement 

<R») * 


Autoincrement Deferred 

<g> (Rrr) . 

Indexed 

(Absolute) 

<ji> A address 

[Rx] 

Displacement 

f B: ( displacement (Rn) 

ir* j “ 


Displacement Deferred 

*{w;} 



n = 0 through 15 
x -• 0 through 14 

Figure 2—Assembler syntax 
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176 

10 

5 

56 

12 

6 


270 


MOVW opcode 

byte displacement mode 
register 5 

d i splacement 

word displacement mode 
register 6 


d isplacement 


Figure 3—MOVW 56 (R5), 270(R7) 



ADDL3 opcode 


r 


1iteral 
\ constant 


mode 

1 


register 
register 


mode 

0 


index prefix 
register 2 


autoincrement 
register 15 (a 


mode 
bsolute) 



Figure 4 —ADDL3 #1, RO, @#A[R2] 
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Figure 5—RSB 


RSB opcode 


and the programmer rarely needs to be concerned with dis¬ 
placement sizes or whether to choose literal or immediate 
mode. The programmer writes the simple form and assem¬ 
bler chooses the address mode which produces the shortest 
instruction length. 

In order to give a better feeling for the instruction format 
and assembler notation, several examples are given in Fig¬ 
ures 3-5. In Figure 3 is an instruction which moves a word 
from an address which is 56 plus the contents of R5 to an 
address which is 270 plus the contents of R6. Note, that the 
displacement 56 is representable in a byte while the displace¬ 
ment 270 requires a word. The instruction occupies 6 bytes. 
In Figure 4 is an instruction which adds 1 to a longword in 
R0 and stores the result at a memory address which is the 
sum of A and 4 times the contents of R2. This instruction 
occupies 9 bytes. Finally, in Figure 5 is a return from sub¬ 
routine instruction. It has no explicit operands and occupies 
a single byte. 

The only significant instance where there is non-general 
specification of operands is in the specification of targets for 
branch instructions. Since invariably the target of a branch 
instruction is a small displacement from the current PC, 
most branch instructions simply take a one byte PC relative 
displacement. This is exactly as if byte displacement mode 
were used with the PC used as the register, except that the 
operand specifier byte is not needed. Because of the per¬ 
vasiveness of branch instructions in code, this one byte 
saving results in a non-trivial reduction in code size. An 
example of the branch instruction branch on equal is given 
in Figure 6. 


Instruction set 

A major goal of the VAX-11 instruction set design was to 
provide for effective compiler generated code. Four deci¬ 
sions helped to realize this goal: 

1. A very regular and consistent treatment of operators. 
Thus, for example, since there is a divide longword 
instruction, there are also divide word and divide byte 
instructions. 


BEQL opcode 


displacement 



2. An avoidance of instructions unlikely to be generated 
by a compiler. 

3. Inclusion of several forms of common operators. For 
example the integer add instructions are included in 
three forms: (a) one operand where the value one is 
added to an operand, (b) two operands where one op¬ 
erand is added to a second, and (c) three operands 
where one operand is added to a second and the result 
stored in a third. Since the VAX-11 instruction format 
allows fully general specifications of the operands, 
VAX-11 programs often have the structure (though not 
the encoding) of the canonic program form proposed 
in Reference 5. 

4. Replacement of common instruction sequences with 
single instructions. Examples of this include procedure 
calling, multiway branching, loop control, and array 
subscript calculation. 

The effect of these decisions is reflected in several obser¬ 
vations. First, despite the larger virtual address and instruc¬ 
tion set support for more data types, compiler (and hand) 
generated code for VAX-11 is typically smaller than the 
equivalent PDP-11 code for algorithms operating on data 
types supported by the PDP-11. Second, of the 243 instruc¬ 
tions in the instruction set about 75 percent are generated 
by the VAX-11 FORTRAN compiler. Of the instructions 
not generated, most operate on data types not part of the 
FORTRAN language. 

A complete list of the VAX-11 instructions is given in the 
Appendix. The following gives an overview of the instruc¬ 
tion set. 

1. Integer logic and arithmetic —Byte, word, and long¬ 
word are the primary data types. A fairly conven¬ 
tional group of arithmetic and logical instructions is 
provided. The result generating dyadic arithmetic and 
logical instructions are provided in two and three op¬ 
erand forms. A number of optimizations are included: 
clear which is a move of zero; test which is a compare 
against zero; and increment and decrement which are 
an optimization of add one and subtract one respec¬ 
tively. A complete set of converts is provided which 
covers both the integer and the floating data types. In 
contrast to other architectures only a few shift type 
instructions are provided: it was felt that shifts are 
mostly used for field isolation which is much more 
conveniently done with the field instructions described 
later. In order to support greater than longword pre¬ 
cision integer operations, a few special instructions are 
provided: extended multiply and divide and add with 
carry and subtract with carry. 

2. Floating point instructions —Again a conventional 
group of instructions are included with result producing 
dyadic operators in two and three operand forms. Sev¬ 
eral specialized floating point instructions are included. 
The extended modulus instruction multiplies two float¬ 
ing operands together and stores the integer and frac¬ 
tion parts of the product in separate result operands. 
The polynomial instruction computes a polynomial 
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from a table of coefficients in memory. Both these 
instructions employ greater than normal precision and 
are useful in high accuracy mathematical routines. A 
convert rounded instruction is provided which imple¬ 
ments the ALGOL rather than FORTRAN conventions 
for converting from floating point to integer. 

3. Address instructions —The move address instructions 
store in the result operand the effective address of the 
source operand. The push address optimizations push 
on the stack (defined by SP) the effective address of 
the source operand. The latter are used extensively in 
subroutine calling. 

4. Field instructions —The extract field instructions ex¬ 
tract a 0 to 32-bit field, sign- or zero-extend it if it is 
less than 32 bits, and store it in a longword operand. 
The compare field instructions compare a (sign- or 
zero-extended if necessary) field against a longword 
operand. The find first instructions find the first oc¬ 
currence of a set or clear bit in a field. 

5. Control instructions —There is a complete set of con¬ 
ditional branches supporting both a signed and, where 
appropriate, an unsigned interpretation of the various 
data types. These branches test the condition codes 
and take a one byte PC relative branch displacement. 
There are three unconditional branch instructions: the 
first taking a one byte PC relative displacement, the 
second taking a word PC relative displacement, and 
the third—called jump—taking a general operand spec¬ 
ification. Paralleling these three instructions are three 
branch to subroutine instructions. These push the cur¬ 
rent PC on the stack before transferring control. The 
single byte return from subroutine instruction returns 
from subroutines called by these instructions. There is 
a set of branch on bit instructions which branch on the 
state of a single bit and, depending on the instruction, 
set, clear, or leave unchanged that bit. 

The add compare and branch instructions are used 
for loop control. A step operand is added to the loop 
control operand and the sum compared against a limit 
operand. The result of the comparison determines 
whether the branch is taken. The sense of the compar¬ 
ison is based on the sign of the step operand. Optimi¬ 
zations of loop control include the add one and branch 
instructions which assume a step of one and the sub¬ 
tract one and branch instructions which assume a step 
of minus one and a limit of zero. 

The case instructions implement the computed go to 
in FORTRAN and case statements in other languages. 
A selector operand is checked to see that it lies in 
range and is then used to select one of table of PC 
relative branch displacements following the instruc¬ 
tion. 

6. Queue instructions —The queue representation is a 
doubly linked circular list. Instructions are provided to 
insert an item into a queue or to remove an item from 
a queue. 

7. Character string instructions —The general move char¬ 
acter instruction takes five operands specifying the 
lengths and starting addresses of the source and des¬ 


tination strings and a fill character to be used if the 
source string is shorter than the destination string. The 
instruction functions correctly regardless of string ov¬ 
erlap. An optimized move character instruction as¬ 
sumes the string lengths are equal and takes three op¬ 
erands. Paralleling the move instructions are two 
compare character instructions. The move translated 
characters instruction is similar to the general move 
character instruction except that the source string 
bytes are translated by a translation table specified by 
the instruction before being moved to destination 
string. The move translated until escape instruction 
stops if the result of a translation matches the escape 
character specified by one of its operands. The locate 
and skip character instructions find respectively the 
first occurence or non-occurence of a character in a 
string. The scan and span instructions find respectively 
the first occurence or non-occurence of a character 
within a specified character set in a string. The match 
characters instruction finds the first occurence of a 
substring within a string which matches a specified 
pattern string. 

8. Packed decimal instructions —A conventional set of 
arithmetic instructions is provided. The arithmetic shift 
and round instruction provides decimal point scaling 
and rounding. Converts are provided to and from long¬ 
word integers, leading separate decimal strings, and 
trailing embedded decimal strings. A comprehensive 
edit instruction is included. 


VAX-11 procedure instructions 

A major goal of the VAX-11 design was to have a single 
system wide procedure calling convention which would 
apply to all inter-module calls in the various languages, calls 
for operating system services, and calls to the common run 
time system. Three VAX-11 instructions support this con¬ 
vention: two call instructions which are indistinguishable as 
far as the called procedure is concerned and a return instruc¬ 
tion. 

The call instructions assume that the first word of a pro¬ 
cedure is an entry mask which specifies which registers are 
to be used by the procedure and thus need to be saved. 
(Actually only R0-R11 are controlled by the entry mask and 
bits 15:12 of the mask are reserved for other purposes.) 
After pushing the registers to be saved on the stack, the call 
instruction pushes AP, FP, PC, a longword containing the 
PSW and the entry mask, and a zero valued longword which 
is the initial value of a condition handler address. The call 
instruction then loads FP with the contents of SP and AP 
with the argument list address. The appearance of the stack 
frame after the call is shown in the upper part of Figure 7. 

The form of the argument list is shown in the lower part 
of Figure 7. It consists of an argument count and list of 
longword arguments which are typically addresses. The 
CALLG instruction takes two operands: one specifying the 
procedure address and the other specifying the address of 
the argument list assumed arbitrarily located in memory. 




974 


National Computer Conference, 1978 




The CALLS instruction also takes two operands: one the 
procedure address and the other an argument count. CALLS 
assumes that the arguments have been pushed on the stack 
and pushes the argument count immediately prior to saving 
the registers controlled by the entry mask. It also sets bit 13 
of the saved entry mask to indicate a CALLS instruction 
was used to make the call. 

The return instruction uses FP to locate the stack frame. 
It loads SP with the contents of FP and restores PSW 
through PC by popping the stack. The saved entry mask 
controls the popping and restoring of Rll through RO. Fi¬ 
nally if the bit indicating CALLS was set, the argument list 
is removed from the stack. 


Memory management design alternatives 

Memory management comprises the mechanisms used (1) 
to map the virtual addresses generated by processes to phys¬ 
ical memory addresses, (2) to control access to memory 
(i.e., to control whether a process has read, write, or no 
access to various areas of memory), and (3) to allow a 
process to execute even if all of its virtual address space is 
not simultaneously mapped to physical memory (i.e., to 
provide so called virtual memory facilities). The memory 
management proved to be the most difficult part of the 
architecture to design. Three alternatives were pursued and 
full designs were completed for the first two alternatives and 
nearly completed for the third. The three alternatives* were: 

1. A paged form of memory management with access 
control at the page level and a small number (4) of 
hierarchical access modes whose use would be dedi¬ 
cated to specific purposes. This represented an evo¬ 
lution of the PDP-11/70 memory management. 

2. A paged and segmented form with access control at 
the segment level and a larger number (8) of hierarchi¬ 
cal access modes which would be used quite generally. 
Although it differed in a number of ways, the design 
was motivated by the Multics 6,7 architecture and the 
Honeywell 6180 implementation. 

3. A capabilities 8,9 form with access control provided by 
the capabilities and the ability to page larger objects 
described by the capabilities. 

The first alternative was finally selected. The second al¬ 
ternative was rejected because it was felt that the real in¬ 
crease in functionality provided inadequately offset the in¬ 
creased architectural complexity. The third alternative 
appeared to offer functionality advantages that could be 
useful over the longer term. However, it was unlikely that 
these advantages could be exploited in the near term. Fur¬ 
ther it appeared that the complexity of the capabilities design 
was inappropriate for a minicomputer system. 

Memory mapping 

The 4.3 gigabyte virtual address space is divided into four 
regions as shown in Figure 8. The first two regions—the 
program and control regions—comprise the per process vir¬ 
tual address space which is uniquely mapped for each proc¬ 
ess. The second two regions—the system region and a region 
reserved for future use—comprise the system virtual address 
space which is singly mapped for all processes. 

Each of the regions serves different purposes. The program 
region contains user programs and data and the top of the 
region is a dynamic memory allocation point. The control 


* It should not be construed that memory management is independent of the 
rest of the architecture. The various memory management alternatives re¬ 
quired different definitions of the addressing modes and different instruction 
level support for addressing. 
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region contains operating system data structures specific to 
the process and the user stack. The system region contains 
procedures which are common to all processes (such as 
those that comprise the operating system and RMS) and (as 
will be seen later) page tables. 

A virtual address has the structure shown in the upper 
part of Figure 9. Bits 8:0 specify a byte within a 512 byte 
page which is the basic unit of mapping. Bits 29:9 specify 
a virtual page number (VPN). Bits 31:30 select the virtual 
address region. The mechanism of mapping consists of using 
the region select bits to select a page table which consists 
of page table entries (PTEs). After a check that it is not too 
large, the VPN is used to index into the page table to select 
a PTE. The PTE contains either (1) 21-bit physical page 
frame number which is concatenated with the nine low order 
byte in page bits to form a 30-bit physical address shown in 
the lower part of Figure 9, or (2) an indication that the 
virtual page accessed is not in physical memory. The latter 
case is called a page fault. Instruction execution in the 
current procedure is suspended and control is transferred to 
an operating system procedure which will cause the missing 
virtual page to be brought into physical memory. At this 
point instruction execution in the suspended procedure can 
resume transparently. 
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Figure 9—Virtual and physical addresses 


The page table for the system region is defined by the 
system base register which contains the physical address of 
the start of the system region page table and the system 
length register which contains the length of the table. Thus 
the system page table is contiguous in physical memory. 

The per process space page tables are defined similarly 
by the program and control region base registers and length 
registers. However, the base registers do not contain phys¬ 
ical addresses: rather, they contain system region virtual 
addresses. Thus the per process page tables are contiguous 
in the system region virtual address space and are not nec¬ 
essarily contiguous in physical memory. This placement of 
the per process page tables permits them to be paged and 
avoids what would otherwise be a serious physical memory 
allocation problem. 

Access control 

At a given point in time a process executes in one of four 
access modes. The modes from most privileged to least are 
called kernel, executive, supervisor and user. The use of 
these modes by VAX/VMS is as follows: 

1. Kernel—Interrupt and exception handling, scheduling, 
paging, physical I/O, etc. 

2. Executive—Logical I/O as provided by RMS. 

3. Supervisor—The command interpreter. 

4. User—User procedures and data. 

The accessibility of each page (read, write, or no access) 
from each access mode is specified in the PTE for that page. 
Any attempt to improperly access a page is suppressed and 
control is transferred to an operating system procedure. The 
accessibility is assumed hierarchically ordered: if a page is 
writable from any given mode, it is also readable; and if a 
page is accessible from a less privileged mode, it is acces¬ 
sible from a more privileged mode. Thus, for example, a 
page can be readable and writable from kernel mode, only 
readable from executive mode, and inaccessible from su¬ 
pervisor and user modes. 

A procedure executing in a less privileged mode often 
needs to call a procedure which executes in a more privi¬ 
leged mode: e.g., a user program needs an operating system 
service performed. The access mode is changed to a more 
privileged mode by executing a change mode instruction 
which transfers control to a routine executing at the new 
access mode. A return is made to original access mode by 
executing a return from exception or interrupt instruction 
(REI). 

The current access mode is stored in the processor status 
longword (PSL) whose low order 16 bits comprise the PSW. 
Also stored in the PSL is the previous access mode; i.e., 
the access mode from which the current access mode was 
called. The previous mode information is used by the special 
probe instructions which validate arguments passed in cross 
access mode calls. 

Procedures running at each of the access modes require a 







976 


National Computer Conference, 1978 


separate stacks with appropriate accessibility. To facilitate 
this, each process has four copies of SP which are selected 
according to the current access mode field in the PSL. A 
procedure always accesses the correct stack by using R14. 

In an earlier section, it was stated that the VAX-11 stand¬ 
ard CALL instruction is used for all calls including those for 
operating system services. Indeed procedures do call the 
operating system using the CALL instruction. The target of 
the CALL instruction is the minimal procedure consisting 
of an entry mask, a change mode instruction, and a return 
instruction. Thus access mode changing is transparent to the 
calling procedure. 

Interrupts and exceptions 

Interrupts and exceptions are forced changes in control 
flow other than that explicitly indicated by the executing 
program. The distinction between them is that interrupts are 
normally unrelated to the currently executing program while 
exceptions are a direct consequence of program execution. 
Examples of interrupt conditions are status changes in I/O 
devices while examples of exception conditions are arith¬ 
metic overflow or a memory management access control 
violation. 

VAX-11 provides a 31 priority level interrupt system. 
Sixteen levels (16-31) are provided for hardware while 15 
levels (1-15) are provided for software. (Level 0 is used for 
normal program execution.) The current interrupt priority 
level (IPL) is stored in a field in the PSL. When an interrupt 
request is made at a level higher than IPL, the current PC 
and PSL are pushed on the stack and new PC obtained from 
a vector selected by the interrupt requester (a new PSL is 
generated by the CPU). Interrupts are serviced by routines 
executing with kernel mode access control. Since interrupts 
are appropriately serviced in a system wide rather than a 
specific process context, the stack used for interrupts is 
defined by another stack pointer called the interrupt stack 
pointer. (Just as for the multiple stack pointers used in 
process context, an interrupt routine accesses the interrupt 
stack using R14.) An interrupt service is terminated by ex¬ 
ecution of an REI instruction which loads PC and PSL from 
the top two longwords on the stack. 

Exceptions are handled like interrupts except for the fol¬ 
lowing: (1) since exceptions arise in a specific process con¬ 
text, the kernel mode stack for that process is used to store 
PC and PSL and (2) additional parameters (such as the 
virtual address causing a page fault) may be pushed on the 
stack. 


Process context switching 

From the standpoint of the VAX-11 architecture, the proc¬ 
ess state or context consists of: 

1. The 15 general registers R0-R13 and R15. 

2. Four copies of R14 (SP): one for each of kernel, ex¬ 
ecutive, supervisor, and user access modes. 


3. The PSL. 

4. Two base and two limit registers for the program and 
control region page tables. 

This context is gathered together in a data structure called 
a process control block (PCB) which normally resides in 
memory. While a process is executing, the process context 
can be considered to reside in processor registers. To switch 
from one process to another it is required that the process 
context from the previously executing process be saved in 
its PCB in memory and the process context for the process 
about to be executed to be loaded from its PCB in memory. 
Two VAX-11 instructions support context switching. The 
save process context instruction saves the complete process 
context in memory while the load process context instruc¬ 
tion loads the complete process context from memory. 


Much like the PDP-11, VAX-11 has no specific I/O in¬ 
structions. Rather, I/O devices and device controllers are 
implemented with a set of registers which have addresses in 
the physical memory address space. The CPU controls I/O 
devices by writing these registers; the devices return status 
by writing these registers and the CPU subsequently reading 
them. The normal memory management mechanism controls 
access to I/O device registers and a process having a partic¬ 
ular device’s registers mapped into its address space can 
control that device using the regular instruction set. 

Compatibility mode 

As mentioned in the VAX-11 overview, compatibility 
mode in the VAX-11 architecture provides the basic PDP- 
11 instruction set less privileged and floating point instruc¬ 
tions. Compatibility mode is intended to support a user as 
opposed to an operating system environment. Normally a 
compatibility mode program is combined with a set of native 
mode procedures whose purpose is to map service requests 
from some particular PDP-11 operating system environment 
into VAX/VMS services. 

In compatibility mode the 16-bit PDP-11 addresses are 
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Figure 10—VAX-11/780 system 
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zero-extended to 32-bits where standard native mode map¬ 
ping and access control apply. The eight 16-bit PDP-11 gen¬ 
eral registers overmap the native mode general registers R0- 
R6 and R15 and thus the PDP-11 processor state is contained 
wholly within the native mode processor state. 

Compatibility mode is entered by setting the compatibility 
mode bit in the PSL. Compatibility mode is left by executing 
a PDP-11 trap instruction (such as used to make operating 
service requests), and on interrupts and exceptions. 


VAX-11/780 IMPLEMENTATION 
VAX-111780 

The VAX-11/780 computer system is the first implemen¬ 
tation of the VAX-11 architecture. For instructions executed 
in compatibility mode, the VAX-11/780 has a performance 
comparable to the PDP-11/70. For instructions executed in 
native mode, the -11/780 has a performance in excess of the 
-11/70 and thus represents the new high end of the -11 (LSI- 
11, PDP-11, VAX-11) family. 

A block diagram of the -11/780 system is given in Figure 
10. The system consists of a central processing unit (CPU), 
the console subsystem, the memory subsystem, and the I/O 
subsystem. The CPU and the memory and I/O subsystems 
are joined by a high speed synchronous bus called the Syn¬ 
chronous Backplane Interconnect (SBI). 

CPU 

The CPU is a microprogrammed processor which imple¬ 
ments the native and compatibility mode instruction sets, 
the memory management, and the interrupt and exception 
mechanisms. The CPU has 32-bit main data paths and is 
built almost entirely of conventional Shottky TTL compo¬ 
nents. 

To reduce effective memory access time the CPU includes 
an 8K byte write through cache or buffer memory. The 
cache organization is 2-way associative with an 8-byte block 
size. To reduce delays due to writes, the CPU includes a 
write buffer. The CPU issues the write to the buffer and the 
actual memory write takes place in parallel with other CPU 
activity. 

The CPU contains a 128 entry address translation buffer 
which is a cache of recent virtual to physical translations. 
The buffer is divided into two 64 entry sections: one for the 
per process regions and one for the system region. This 
division facilitates permitting the system region translations 
to remain unaffected by a process context switch. 

A fourth buffer in the CPU is the 8-byte instruction buffer. 
It serves two purposes. First, it decomposes the highly var¬ 
iable instruction format into its basic components and, sec¬ 
ond, it constantly fetches ahead to reduce delays in obtaining 
the instruction components. 

The CPU includes two standard clocks. The program¬ 
mable real-time clock is used by the operating system for local 


timing purposes. The time-of-year clock with its own battery 
backup is the long term time reference for the operating 
system. It is automatically read on system startup to elimi¬ 
nate the need for manual entry of data and time. 

The CPU includes 12K bytes of writable diagnostic con¬ 
trol store (WDCS) which is used for diagnostic purposes, 
implementation of certain instructions, and for future micro¬ 
code changes. As an option for very sophisticated users, 
another 12K bytes of writable control store is available. 

A second option is the floating point accelerator (FPA). 
Although the basic CPU implements the full floating point 
instruction set, the FPA provides high speed floating point 
hardware. It is logically invisible to programs and only af¬ 
fects their running time. 

Console subsystem 

The console subsystem is centered around an LSI-11 com¬ 
puter with 16K bytes of RAM and 8K bytes of ROM (used 
to store the LSI-11 bootstrap, LSI-11 diagnostics, and con¬ 
sole routines). Also included are a floppy disk, an interface 
to the console terminal, and a port for remote diagnostic 
purposes. 

The floppy disk in the console subsystem serves multiple 
purposes. It stores the main system bootstrap and diagnos¬ 
tics and serves as a medium for distribution of software 
updates. 

SBI 

The SBI is the primary control and data transfer path in 
the -11/780 system. Because the cache and write buffer 
largely decouple the CPU performance from the memory 
access time, the SBI design was optimized for bandwidth 
and reliability rather than the lowest possible access time. 

The SBI is a synchronous bus with a cycle time of 200 
nsec. The data path width of the SBI is 32 bits. During each 
200 nsec cycle either 32 bits of data or a 30-bit physical 
address can be transferred. Since each 32-bit read or write 
requires transmission of both address and data, two SBI 
cycles are used for a complete transaction. The SBI protocol 
permits 64-bit reads or writes using one address cycle and 
two data transfer cycles: the CPU and I/O subsystem use 
this mode whenever possible. For read transactions the bus 
is reacquired by the memory in order to send the data: thus 
the bus is not held during the memory access time. 

Arbitration of the SBI is distributed: each interface to the 
SBI has a specific priority and its own bus request line. 
When an interface wishes to use the bus, it asserts its bus 
request line. If at the end of a 200 nsec cycle there are no 
interfaces of higher priority requesting the bus, the interface 
takes control of the bus. 

Extensive checking is done on the SBI. Each transfer is 
parity checked and confirmed by the receiver. The arbitra¬ 
tion process and general observance of the SBI protocol are 
checked by each SBI interface during each SBI cycle. The 
processor maintains a running 16-cycle history of the SBI: 
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any SBI error condition causes this history to be locked and 
preserved for diagnostic purposes. 

Memory subsystem 

The memory subsystem consists of one or two memory 
controllers with up to 1M bytes of memory on each. The 
memory is organized in 64-bit quadwords with an 8-bit ECC 
which provides single bit error correction and double bit 
error detection. The memory is built of 4K MOS RAM 
components. 

The memory controllers have buffers which hold up to 
four memory requests. These buffers substantially increase 
the utilization of the SBI and memory by permitting the 
pipelining of multiple memory requests. If desired, quad- 
word physical addresses can be interleaved across the mem¬ 
ory controllers. 

As an option, battery backup is available which preserves 
the contents of memory across short term power failures. 

I/O subsystem 

The I/O subsystem consists of buffered interfaces or 
adapters between the SBI and the two types of peripheral 
busses used on PDP-11 systems: the Unibus and the Mass- 
bus. One Unibus adapter and up to four Massbus adapters 
can be configured on a VAX-11/780 system. 

The Unibus is a medium speed multiplexor bus which is 
used as a primary memory as well as peripheral bus in many 
PDP-11 systems. It has an 18-bit physical address space and 
supports byte and word transfers. In addition to imple¬ 
menting the Unibus protocol and transmitting interrupts to 
the CPU, the Unibus adapter provides two other functions. 
The first is mapping 18-bit Unibus addresses to 30-bit SBI 
physical addresses. This is accomplished in a manner sub¬ 
stantially identical to the virtual to physical mapping imple¬ 
mented by the CPU. The Unibus address space is divided 
into 512 512-byte pages. Each Unibus page has a page table 
entry (residing in the Unibus adapter) which maps addresses 
in that page to physical memory addresses. In addition to 
providing address translation, the mapping permits contig¬ 
uous transfers on the Unibus which cross page boundaries 
to be mapped to discontiguous physical memory page 
frames. 

The second function performed by the Unibus adapter is 
assembling 16-bit Unibus transfers (both reads and writes) 
into 64-bit SBI transfers. This operation (which is applicable 
only to block transfers such as from disks) appreciably re¬ 
duces SBI traffic due to Unibus operations. There are 15 8- 
byte buffers in the Unibus adapter permitting 15 simulta¬ 
neous buffered transactions. Additionally there is a un-buf- 
fered path through the Unibus adapter permitting an arbitrary 
number of simultaneous un-buffered transfers. 

The Massbus is a high speed block transfer bus used 
primarily for disks and tapes. The Massbus adapter provides 
much the same functionality as the Unibus adapter. The 
physical addresses into which transfers are made are defined 


by a page table: again this permits contiguous device trans¬ 
fers into discontiguous physical memory. 

Buffering is provided in the Massbuss adapter which min¬ 
imizes the probability of device overruns and assembles data 
into 64-bit units for transfer over the SBI. 
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APPENDIX—VAX-11 INSTRUCTION SET 

Integer and Floating Point Logical Instructions 

MOV- Move(B,W,L,F,D,Q)* 

MNEG- Move Negated(B,W,L,F,D) 

MCOM- Move Complemented(B,W,L) 

MOVZ- Move Zero-Extended(BW,BL,WL) 

CLR- Clear(B,W,L=F,Q=D) 

CVT— Convert(B,W,L,F,D)(B,W,L,F,D) 

CVTR-L Convert Rounded(F,D) to Longword 

CMP- Compare(B,W,L,F,D) 

TST- Test(B,W,L,F,D) 

BIS-2 Bit Set(B,W,L)2-Operand 

BIS-3 Bit Set(B,W,L)3-Operand 

BIC-2 Bit Clear(B,W,L)2-Operand 

* B = byte, W = word, L - longword, F = floating, D = double floating, Q 
= quadword, S = set, C — clear. 
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BIC-3 

Bit Clear(B,W,L)3-Operand 

BIT- 

Bit Test(B,W,L) 

XOR-2 

Exclusive OR(B,W,L)2-Operand 

XOR-3 

Exclusive OR(B,W,L)3-Operand 

ROTL 

Rotate Longword 

PUSHL 

Push Longword 


Integer and Floating Point Arithmetic Instructions 

INC- 

Increment(B, W, L) 

DEC- 

Decrement(B, W, L) 

ASH- 

Arithmetic Shift(L,Q) 

ADD-2 

Add(B,W,L,F,D)2-Operand 

ADD-3 

Add(B,W,L,F,D)3-Operand 

ADWC 

Add with Carry 

ADAWI 

Add Aligned Word Interlocked 

SUB-2 

Subtract(B,W,L,F,D)2-Operand 

SUB-3 

Subtract( B, W ,L, F, D) 3-Operand 

SBWC 

Subtract with Carry 

MUL-2 

Multiply(B,W,L,F,D)2-Operand 

MUL-3 

Multiply (B, W, L, F, D) 3-Operand 

EMUL 

Extended Multiply 

DIV-2 

Divide(B,W,L,F,D)2-Operand 

DIV-3 

Divide(B, W ,L ,F ,D)3-Operand 

EDIV 

Extended Divide 

EMOD- 

Extended Modulus(F,D) 

POLY- 

Polynomial Evaluation (F,D) 


Index Instruction 


INDEX Compute Index 


Packed Decimal Instructions 

MOVP Move Packed 

CMPP3 Compare Packed 3-Operand 

CMPP4 Compare Packed 4-Operand 

ASHP Arithmetic Shift Round and Packed 

ADDP4 Add Packed 4-Operand 

ADDP6 Add Packed 6-Operand 

SUBP4 Subtract Packed 4-Operand 

SUBP6 Subtract Packed 6-Operand 

MULP Multiply Packed 

DIVP Divide Packed 

CVTLP Convert Long to Packed 

CVTPL Convert Packed to Long 

CVTPT Convert Packed to Trailing 

CVTTP Convert Trailing to Packed 

CVTPS Convert Packed to Separate 

CVTSP Convert Separate to Packed 

EDITPC Edit Packed to Character String 


Character String Instructions 

MOVC3 Move Character 3-Operand 

MOVC5 Move Character 5-Operand 

MOVTC Move Translated Characters 

MOVTUC Move Translated Until Character 
CMPC3 Compare Characters 3-Operand 

CMPC5 Compare Characters 5-Operand 

LOCC Locate Character 

SKPC Skip Character 

SCANC Scan Characters 

SPANC Span Characters 

MATCHC Match Characters 


Variable-Length Bit Field Instructions 

EXTV Extract Field 

EXTZV Extract Zero-Extended Field 

INSV Insert Field 

CMPV Compare Field 

CMPZV Compare Zero-Extended Field 

FFS Find First Set 

FFC Find First Clear 


Branch on Bit Instructions 

BLB- Branch on Low B(S,C1) 

BB- Branch on Bit(S,Cl) 

BBS- Branch on Bit Set and(S,Cl)Bit 

BBC Branch on Bit Clear and(Set,lear)Bit 

BBSSI Branch on Bit Set and Set Bit Interlocked 

BBCCI Branch on Bit Clear and Clear Bit Inter¬ 

locked 


Queue Instructions 

INSQUE Insert Entry in Queue 

REMQUE Remove Entry from Queue 


Address Manipulation Instructions 

MOV A- Move Address(B,W,L=F,Q=D) 

PUSHA- Push Address(B,W,L=F,Q=D)on Stack 


Processor State Instructions 

PUSHR Push Registers on Stack 

POPR Pop Registers from Stack 

MOVPSL Move from Processor Status Longword 

BISPSW Bit Set Processor Status Word 
BICPSW Bit Clear Processor Status Word 




980 


National Computer Conference, 1978 


Unconditional Branch and Jump Instructions 

Subroutine Call and Return Instructions 



BSB 

Branch to Subroutine with(B,W)Displacement 

BR- 

Branch with(B,W)Displacement 

JSB 

Jump to Subroutine 

JMP 

Jump 

RSB 

Return from Subroutine 

Branch on Condition Code 

Procedure Call and Return Instructions 



CALLG 

Call Procedure with General Argument List 

BUSS 

Less Than 

CALLS 

Call Procedure with Stack Argument List 

BLSSU 

Less Than Unsigned 

RET 

Return from Procedure 

(BCS) 

(Carry Set) 



BLEQ 

Less Than or Equal 



BLEQU 

Less Than or Equal Unsigned 

Access Mode Instructions 

BEQL 

Equal 



(BEQLU) 

(Equal Unsigned) 

CHM 

Change Mode to 

BNEQ 

Not Equal 


(Kernel, Executive Supervisor,User) 

(BNEQU) 

(Not Equal Unsigned) 

REI 

Return from Exception or Interrupt 

BGTR 

Greater Than 

PROBER 

Probe Read 

BGTRU 

Greater Than Unsigned 

PROBEW 

Probe Write 

BGEQ 

Greater Than or Equal 



BGEQU 

Greater Than or Equal Unsigned 



(BCC) 

(Carry Clear) 

Privileged Processor Register Control Instructions 

BVS 

Overflow Set 



BVC 

Overflow Clear 

SVPCTX 

Save Process Context 



LDPCTX 

Load Process Context 



MTPR 

Move to Process Register 

Loop and Case Branch 

MFPR 

Move from Processor Register 

ACB- 

Add, Compare and Branch(B,W,L,F,D) 

Special Function Instructions 

AOBLEQ 

Add One and Branch Less Than or Equal 



AOBLSS 

Add One and Branch Less Than 

CRC 

Cyclic Redundancy Check 

SOBGEQ 

Subtract One and Branch Greater Than or 

BPT 

Breakpoint Fault 


Equal 

XFC 

Extended Function Call 

SOBGTR 

Subtract One and Branch Greater Than 

NOP 

No Operation 

CASE- 

Case on(B,W,L) 

HALT 

Halt 
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INTRODUCTION 

PEPE (Parallel Element Processing Ensemble) was de¬ 
signed, developed, and produced for the U.S. Army Ballistic 
Missile Defense Advanced Technology Center (BMDATC) 
for research on Ballistic Missile Defense (BMD) systems. 
With its host computer, the CDC 7600, it is potentially the 
world’s most powerful computer system. PEPE employs 
parallel processing and associative data-access techniques, 
and while it can operate in a stand alone mode, it is designed 
primarily to augment more conventional computers in BMD 
service. Potential applications besides BMD for which the 
machine is adapted include weather forecasting, air traffic 
control, image data processing, signal processing, and other 
applications exhibiting inherent parallelism and requiring 
extensive computational power. 

The PEPE system can be considered simply as a large 
master computer, called a host, controlling many smaller 
slave processors, called elements. In the present design, the 
host is a CDC 7600, and there are 288 elements. Each ele¬ 
ment comprises three processors sharing a common data 
memory. One of these processors, called a correlation unit, 
is used for inputting data and has an instruction repertoire 
especially suited for the rapid association of new data with 
data already on file. The second processor, called the arith¬ 
metic unit, has a repertoire similar to that featured in con¬ 
ventional high-power general-purpose machines; i.e., fixed 
and floating point arithmetic operations, load and store, and 
logical operations. The third processor, called the associa¬ 
tive output unit, is used for ordering and outputting data and 
is especially designed to perform complex, multidimensional 
file searches rapidly and efficiently. Each of the three pro¬ 
cessors is driven by its own control unit, which simultane¬ 
ously drives all of the corresponding processors in the en¬ 
semble of elements. The three control units are also capable 
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of executing their own sequential programs. They are com¬ 
bined into a control console, which drives the ensemble of 
elements in parallel and interfaces the ensemble with the 
host. The complete PEPE/Host system, then, is a multipro¬ 
cessor employing seven processor types in all (host, three 
sequential processors, and three parallel processors). All 
seven processor types are capable of simultaneous, over¬ 
lapped operation. 

Software for the PEPE includes the compilers and assem¬ 
blers for the seven PEPE processors and a linkage editor for 
binding programs into executable load modules. The entire 
machine can be programmed in a single language called 
PFOR, which is a superset of FORTRAN. System software 
also includes an instruction-level simulator for PEPE, a gen¬ 
eral-purpose real-time operating system, and a general util¬ 
ities package. 

The PEPE program was a complete system effort, requir¬ 
ing problem analysis; generation of a hierarchy of system, 
functional, and implementation specifications; hardware de¬ 
sign, development, production, and test; system support 
software design, development, production and test; tacti¬ 
cal/problem software design, development, and test; and 
integration into a total real-time systems environment em¬ 
ploying other computers, missiles, radars, and command, 
communications, and control networks. All of these activ¬ 
ities were carried on concurrently. Moreover, much of the 
hardware architecture and programming techniques had 
never before been attempted on such a large scale for real¬ 
time service. It was obvious, therefore, that conventional 
procedures for executing a development program of this 
magnitude and in particular developing the PEPE architec¬ 
ture would not be adequate. Such procedures, employing 
the traditional sequence of problem analysis, design, hard¬ 
ware development, hardware fabrication and test, program 
design, development, production and test, and finally system 
test and validation, often result in major system errors being 
discovered in the last phase of the sequence when it is too 
late to do much about them. Because of the complexity of 
the PEPE development program, major system design errors 
would almost certainly occur (under the conventional pro- 


981 



982 


National Computer Conference, 1978 


cedures) and would propagate through the various stages of 
the program to be discovered in the final stage, system val¬ 
idation. To avoid the high risk inherent in the conventional 
procedures, PEPE system designers decided to employ a 
top-down design approach in which system validation 
started early and continued throughout the entire program. 
Thus, the program would be executed in a logical progres¬ 
sion of validated baseline steps. The validation would be 
accomplished through a combination of functional and an¬ 
alytic simulations. Early in the project, the architecture and 
the operating system were validated via functional simula¬ 
tions which modeled the operation of the PEPE in a BMD 
environment employing a variety of attack scenarios. Fur¬ 
ther functional simulations of more complex environments 
employing PEPE in the National BMD Site Defense System 
were completed during the first half of 1975. A 36-element 
version of the PEPE hardware was delivered to the 
BMDATC Advanced Research Center in Huntsville in April 
1976. Analytic simulations, employing the instruction-level 
hardware simulator and the PEPE hardware itself, were 
conducted during the last three years. 

BMD DATA PROCESSING PROBLEM 

Figure 1 portrays a simplified subset of the BMD data 
processing problem; only that part of the total problem as¬ 
sociated with detecting, tracking, and classifying targets in 
the field of view of the radar is considered in this paper. 
The radar generates a pencil beam capable of being pointed, 
within a few microseconds, in any direction within the sur¬ 
veillance volume. Beams are either transmit or receive 
beams; each transmit beam can generate any one of a num¬ 
ber of different pulse configurations (carrier frequency, 
number of pulses, pulse coding, frequency modulation, pulse 
length, etc., can be varied) and each receive beam can 
likewise assume any one of a number of configurations (car¬ 
rier frequency, range extent, matched filter, etc., can be 
varied). Typically, the radar generates several thousand 
transmit beams per second and a proportionate number of 
receive beams. Generally, several hundred transmit and re¬ 
ceive beam pairs are reserved for searching the upper part 
of the surveillance volume; these are generated in a raster 
scan pattern designed so that no object can penetrate the 



volume without being detected by at least one search beam. 
Interleaved at random in both time and space among the 
search beams, as required by the target environment and 
battle situation, are a variety of special beams for verifying 
search detections, precision tracking of previously detected 
and acquired targets, transmitting guidance commands to 
interceptors, target discrimination and classification, and 
several other functions. It is the job of the computer to 
initiate a file on each detected object, associate each radar 
return with its proper file, employ the information in each 
return to update the file, perform complex mathematical 
functions on the files, and generate requests for additional 
radar information. Each request becomes part of the file on 
a particular target. Periodically, the files are searched in 
some sort of priority fashion to generate orders for radar 
pulses. Details on each pulse are formulated individually by 
the computer based on the requests in the target files; those 
requests which are granted are transmitted to the radar in 
the form of orders. The objective is to select, for each pulse, 
that order which will be of most benefit to the defense 
system at that time. The radar transmits the ordered pulses, 
new returns are thereby obtained, and the radar computer 
loop is closed. The foregoing functions must all be per¬ 
formed within rather severe time deadlines. The BMD data 
processing problem, then, can be described as keeping a 
large, complex data base updated in real time. Moreover, 
the means for acquiring the information required for updat¬ 
ing is under control of the computer. 

Considering the above rather simplified description of a 
representative part of the BMD data processing problem, 
one can deduce the types of processing required, and then 
synthesize a machine architecture to satisfy the require¬ 
ments; this was done in the case of the PEPE design. 

First, means must be provided to associate verified radar 
search returns with target data already on file in computer 
memory. Every such return must be compared with all target 
data on file. Each comparison, or association operation, 
requires a forward prediction of all filed target positions to 
the time of the return, calculation of three-dimensional gates 
(gate dimensions may be more or less than 3, depending on 
correlation criteria) about the positions, and a match of the 
radar return against all gate volumes. 

A conventional sequential computer must execute the 
foregoing operations as an mxn number of operations, 
where m is the number of new returns in a given time 
interval and n is the number of targets on file. This means 
that the data processing resources required for correlation 
of new returns is an exponential function of the target traffic. 
Since correlation is one of the major consumers of sequential 
computer resources in BMD service, an architecture that 
performs correlation more efficiently is indicated. Associa¬ 
tive processors perform the correlation function as an mx 1 
number of operations, independent of the number of targets 
on file. Therefore, a computer architecture for BMD service 
could advantageously possess associative data input char¬ 
acteristics. 

Second, lengthy scientific computations must be made on 
a large number of separate, independent data sets, as in 
Kalman filtering of radar returns. Computations are identical 
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for all targets, so that considerable leverage could be ob¬ 
tained by assigning targets to separate, dedicated execution 
units operating on data in dedicated memories, all driven 
simultaneously from a single control unit executing a single 
program. Therefore, a computer architecture for BMD ser¬ 
vice could advantageously possess parallel processing char¬ 
acteristics. 

Third, multidimensional, prioritized file searches must be 
made through target files for radar pulse requests, so that 
the radar pulses can be scheduled efficiently, in observance 
of a variety of constraints, and in accordance with the im¬ 
mediate needs of the instantaneous total battle environment. 
Such searches are made inefficiently in conventionally ad¬ 
dressed memories, but very efficiently in associative mem¬ 
ories. Therefore, a computer architecture for BMD service 
could advantageously employ associative data retrieval and 
output characteristics. Moreover, associative data retrieval 
would be of great assistance in one of the most critical and 
resource-consuming functions which must be performed in 
a total BMD system, a function largely ignored in this paper. 
The function is that of real-time control, or assigning re¬ 
sources and adjusting algorithms, procedures, function ex¬ 
ecution rates, and overall system behavior dynamically and 
optimally in accordance with instantaneous system de¬ 
mands, status, and defense strategy. This function, because 
it is sequential and decision-making in general, is best han¬ 
dled on a conventional sequential computer. However, the 
complex data interrogation functions which are necessary to 
efficient execution of the real-time-control task are greatly 
simplified when the sequential computer has associative ac¬ 
cess to the dynamic system data base. 

In addition to the foregoing three primary BMD data proc¬ 
essing requirements of correlation, scientific computation, 
and multidimensional file search (which are incidentally 
found to a similar degree in other problems), there are ad¬ 
ditional requirements common to military computers in gen¬ 
eral and to BMD computers in particular. First, the com¬ 
puter must be extremely reliable, and this reliability should 
preferably be an inherent characteristic of the architecture. 
High reliability is difficult to achieve in a computer suitable 
for BMD service, because of the very large amount of hard¬ 
ware such computers must contain. The latter is true be¬ 
cause, in the final analysis, great computational power (es¬ 
timates for BMD requirements run from tens to hundreds of 
millions of instructions per second) is achieved only via the 
employment of large amounts of CPU hardware. An archi¬ 
tecture acceptable for BMD service must allow high relia¬ 
bility despite the large amount of hardware required. Parallel 
associative architectures provide opportunities for meeting 
this requirement, so long as the individual elements in the 
parallel array can be kept independent of one another. Then, 
individual elements can fail without affecting others, and 
without affecting the problem solution. Fortunately, many 
of the data sets dealt with in BMD computations are them¬ 
selves independent, and lend themselves to assignment to 
independent processing elements. Since the data sets are 
independent, there is no need for interelement communica¬ 
tion. This permits the arrangement of processing elements 
to be almost completely unstructured, so that no particular 


one or combination of elements is needed for successful 
problem completion. 

A final architectural requirement, important for BMD ap¬ 
plications, is that the data processing system be capable of 
easy expansion to meet increased requirements. A parallel, 
associative architecture, where the number and organiza¬ 
tional structure of processing elements can be arbitrary, 
lends itself to this requirement too. Additional elements 
could be added to handle additional target traffic, and all of 
the programs could be written to be independent of the 
number and/or location of processing elements, so that pro¬ 
grams need not be modified to accommodate hardware ex¬ 
pansion. 

Summarizing, the foregoing requirements led to consid¬ 
eration of a parallel, associative architecture. However, 
such computers which have been designed and/or built in 
the past fall generally into two categories, neither of which 
are really optimized for BMD service. First, past and con¬ 
temporary associative processors are characterized by very 
primitive per-element computational capability (usually bit- 
serial); they achieve high processing speed by virtue of keep¬ 
ing a very large number of elements (say thousands) busy 
simultaneously. Computers for BMD service would not re¬ 
quire the extremely large number of elements possible in 
such machines, but they do require fairly powerful elements 
because of the extensive per-target scientific calculations 
required. The second category is exemplified by the ILLIAC 
IV, which certainly has sufficient per-element computation 
capability, but an insufficient number of elements (tens), 
assuming that one wants to assign one target to an element 
(this is not really necessary, but not to do so leads to other 
complications and is beyond the scope of this paper). More¬ 
over, ILLIAC IV contains features such as interelement 
communication which are not essential for BMD service, 
but which contribute to lower reliability because of the more 
or less rigid structure imposed upon the array by the inter¬ 
communication facilities. It would seem then, that the par¬ 
allel associative architecture specified for BMD service 
should have capabilities somewhere between those offered 
by associative processors and ILLIAC IV. It should have 
a moderately large number of elements (hundreds), and each 
element should have a fairly powerful scientific computation 
capability. Moreover, each element should comprise three 
execution units, one each for input data correlation, parallel 
scientific computation, and associative file search for data 
output. To maximize throughput, all three execution units 
in the elements should be capable of simultaneous operation. 
The elements should have no positional significance; i.e., 
they should be organized into a completely unstructured 
array—an ensemble. Interelement communication and local 
indexing of data, while of considerable value in parallel 
computers designed to operate on interdependent data sets, 
can be eliminated because they are not needed and in fact 
are not even desirable. Finally and perhaps most important, 
the total data processing system should have high through¬ 
put in sophisticated branching and decision-making opera¬ 
tions, but these operations should not reduce throughput on 
the less complex but resource-consuming parallel data proc¬ 
essing functions. This means that the total data processing 
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system should have separate but closely cooperating facili¬ 
ties for both sequential and parallel processing tasks. The 
architecture which evolved from the foregoing considera¬ 
tions, PEPE, is described in the following section. 

PEPE ARCHITECTURAL OVERVIEW 

A system block diagram of the PEPE is shown in Figure 
2. The Host, a CDC 7600, is connected to the three PEPE 
Control Units (collectively, the Control Console) via three 
standard CDC 7600 MUX (Input/Output Multiplexor) chan¬ 
nels, one for each Control Unit. The CDC 7600 then sees 
the PEPE as three independent PPUs (Peripheral Processing 
Units). Each Control Unit is equipped with a modular Host 
Interface Unit; by replacing Interface Units, other interface 
connections with different Host machines are possible. The 
Host provides overall executive control, through its operat¬ 
ing system, of the Host-PEPE configuration. It loads pro¬ 
gram and initial data into the PEPE memories, schedules 
and dispatches appropriate tasks to the PEPE, executes 
those sequential tasks which it does more efficiently than 
the PEPE, and executes utility functions such as compiling 
PEPE programs and reading and writing to peripheral de¬ 
vices. 

The Control Console contains three independent Control 
Units (See Figure 3); they are similar but the Correlation 
Control Unit is optimized for inputting data to the elements, 
the Arithmetic Control Unit is optimized for scientific cal¬ 
culation in the elements, and the Associative Output Control 
Unit is optimized for outputting data from the elements. 
Each Control Unit is a multiprocessor which splits one in¬ 
struction stream into two parts, one sequential which is 
executed within the Control Unit, and one parallel which is 
executed in the ensemble of elements. 

Arithmetic control unit and parallel arithmetic units 

The Arithmetic Control Unit is shown in block-diagram 
form in Figure 4. This unit contains a 32,768-word, 32 bit- 
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Figure 3—PEPE control console 


per-word, random access semiconductor memory with a 
cycle time of 200 ns. A separate data memory contains 2048 
random-access 32-bit words, implemented in ECL with a 
cycle time of 100 ns. The Sequential Control Logic (SCL) 
is a fairly conventional processor containing the usual A 
(Accumulator), B (operand), and Q (quotient/product) reg¬ 
isters, plus 15 index registers. It executes integer (24-bit) 
arithmetic, and logical and branch instructions fetched from 
program memory on data obtained from data memory. Pro¬ 
gram memory contains a mixture of sequential and parallel 
instructions. The SCL executes the sequential instructions 
itself, but passes the parallel instructions, via the Parallel 
Instruction Queue (a conventional instruction stack) to the 
Parallel Instruction Control Unit (PICU). This unit decodes 
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Figure 2—PEPE architecture 


Figure 4 —PEPE arithmetic control unit 
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the parallel instructions and broadcasts the resulting mi¬ 
crooperation signals to all of the Arithmetic Units in the 
ensemble. The Arithmetic Units, provided they are active, 
all execute the same instructions simultaneously on data 
obtained from their own local data memories. The parallel 
instruction set is quite rich, containing the conventional 
arithmetic operations (plus square root) in both integer and 
floating point, load, store, and logical operations. In addi¬ 
tion, the instruction set contains associative instructions 
which activate and deactivate elements based on element 
accumulator content and/or element tag register content. 
Only active elements execute instructions. Each Arithmetic 
Unit can execute about one million instructions per second, 
based on averages over instruction mixes encountered in 
typical scientific calculations. 

Correlation control unit and parallel correlation units 

The Correlation Control Unit (CCU) is similar to the 
Arithmetic Control Unit, but has a smaller, faster program 
memory and no Parallel Instruction Queue. Moreover, its 
PICU does not include a floating-point arithmetic capability, 
nor can it execute sequential floating-point operations. The 
CCU broadcasts a sequential stream of input data from the 
Host, or some other source, to all elements, and enters 
words from the input stream into properly activated ele¬ 
ments. The Correlation Control Unit parallel instruction set 
is rich in associative match and compare operations, so that 
incoming data can be rapidly and efficiently correlated with 
data already in element memory, and thus be placed in the 
proper elements. Instructions are fetched from the CCU 
program memory and decoded, then the resulting microop¬ 
eration signals are broadcast to the Correlation Units of all 
elements. The Correlation Units can be activated and deac¬ 
tivated independently of the element Arithmetic Units; only 
active Correlation Units execute instructions. 

Each Correlation Unit contains a B register and 16 Cor¬ 
relation Registers which support execution of integer and 
logical operations, plus associative match and comparison 
operations among entering data and data retrieved from ele¬ 
ment memory. The Correlation Unit shares element memory 
with the Arithmetic Unit and the Associative Output Unit. 
Since the Correlation Unit does not perform floating point 
operations, its average instruction execution rate is about 
five million instructions per second, much faster than that 
of the Arithmetic Unit. 


Associative output control unit and parallel associative 

output units 

The Associative Output Control Unit (AOCU) is similar 
to the Arithmetic Control Unit, but, like the CCU, has a 
smaller, faster program memory, no Parallel Instruction 
Queue, and no capability for sequential or parallel floating¬ 
point arithmetic operations. Its function is to extract data 
sequentially from properly activated element memories, 
based on associative search instruction sequences which it 


fetches from its own program memory, decodes, and broad¬ 
casts (microoperation signals) to all element Associative 
Output Units (AOU). The AOUs can be activated inde¬ 
pendently of the element Arithmetic Units; only active 
AOUs execute instructions. 

Each AOU contains conventional A and B Registers 
which support operation of integer and logical operations 
plus the aforementioned associative search operations. Par¬ 
allel searches can be based on multidimensional search cri¬ 
teria, and data can be ordered in several different ways as 
they are obtained from the ensemble of elements. As in any 
associative memory, data output is sequential (in PEPE, 
sequential by 32-bit word). Data are buffered into AOCU 
Data Memory before transfer to the Host or some other 
device. Like the CUs, the AOUs execute instructions at an 
average rate of five million instructions per second, and they 
share element memory. 

Internal interfaces 

As shown in Figure 3, three major internal interfaces exist 
within the Control Console. 

First, the Intercommunication Logic provides means for 
transferring data and control words among the control units. 
It contains a real-time clock and an interval timer accessible 
to all control units, and it provides means for the CCU and/ 
or the AOCU to interrupt the ACU. 

Second, the Element Memory Control receives requests 
from the three control units for element memory access. It 
performs any needed conflict resolution and transmits mem¬ 
ory addresses to the elements. 

Third, the Output Data Control is provided to resolve 
conflicts between AU and AOU requests to output data to 
the ACU and AOCU, respectively, over a common output 
data bus. The CUs have no capability for outputting data. 


Design considerations 

Several major design tradeoff decisions were made in as¬ 
signing various functional capabilities to the different PEPE 
machine resources. These tradeoffs were made after exten¬ 
sive experiments involving simulated BMD exercises for a 
variety of attack scenarios and defense configurations. The 
tradeoffs were optimized for BMD service, and are not nec¬ 
essarily the ones that would have been made for PEPE 
application to other problems. 

First, no provisions were made in the PEPE design for 
interelement communication. Data can be moved from ele¬ 
ment to element, but only through the Control Console, and 
the data transfer is sequential, one word at a time. This lack 
of a parallel interelement data transfer capability limits 
PEPE speed on problems involving computation performed 
on interrelated data sets (as encountered, for instance, in 
numerical solutions of partial differential equations), but for 
computations involving independent data sets, it is a real 
advantage. For the latter class of problem, each independent 
data set (for instance, a file on an aircraft, a missile, a 
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person) is assigned to an element, and each element per¬ 
forms identical operations simultaneously on the indepen¬ 
dent files. Since there is seldom, if ever, any interaction 
among the files, no interelement communication is required. 
The hardware simplification achieved by not providing in¬ 
terelement communication is obvious, but there are other 
more important advantages. The ensemble of elements can 
be completely unstructured (accordingly, the collection of 
elements is called an ensemble rather than an array); no 
element has any positional significance. Thus, the elements 
can be accessed, ordered, and grouped for data manipulation 
purposes purely on the basis of their contents, or associa- 
tively. An element can fail, with no impact on the calcula¬ 
tions occurring in the other elements. Moreover, for many 
problems where data are periodically updated on the basis 
of continually arriving new data, only the historical state of 
an independent file is lost when an element fails. New data 
destined for that element are unable to correlate, so the data 
are automatically entered into an empty element to begin a 
new file. Thus, graceful degradation and automatic recovery 
are achieved naturally. Further, elements can be added as 
required with no effect on software or program execution. 
In fact, PEPE programs are oblivious of the number of 
elements, and programmers do not know or care either, as 
long as there are enough. 

Another design decision involved floating-point versus 
fixed-point arithmetic. This decision had to be made for 
each of the six different processor types in PEPE (one se¬ 
quential and one parallel processor for each control unit). 
The sequential processors in the control units are used 
mainly for comparison, branching, control of program flow, 
and supervisory operations, all of which can be executed 
with only integer, branching, load, store, and logical oper¬ 
ations. Therefore, no floating-point hardware was provided 
in the control unit sequential processors. The Arithmetic 
Units in the elements have scientific calculations as their 
primary responsibility; therefore, they were provided with 
a full, powerful, floating-point arithmetic instruction reper¬ 
toire in addition to all of the other conventional and asso¬ 
ciative grouping instructions. The CUs and AOUs are 
mainly required to input/output data and perform associative 
comparisons, matches, searches, and ordering functions, all 
of which can be handled with logical and integer arithmetic 
operations. However, they are occasionally called upon to 
execute short subroutines containing floating-point opera¬ 
tions. Rather than provide expensive floating-point hard¬ 
ware in all of the CUs and AOUs to accommodate the rather 
infrequent requirements, it was decided to include instead 
provision for the CCU and AOCU to interrupt the ACU. 
Then, the ACU could perform the floating-point routines on 
demand for the CCU and the AOCU. A considerable amount 
of hardware was thereby saved, and simulations show that 
the performance degradation caused by lack of floating-point 
capability in the CUs and AOUs is insignificant. 

A third design decision involved the Parallel Instruction 
Queue, which was provided only in the parallel instruction 
stream in the ACU. It was provided there because the ACU 
program memory is much larger and therefore slower than 
the other PEPE memories (200 ns vs 100 ns cycle time). 


Also, the average ACU instruction time is one microsecond, 
so that ACU parallel program execution time could be sig¬ 
nificantly reduced by inserting a fast stack (the PIQ has 16 
ECL registers) between the program memory and the PICU. 
Since the CCU/CU and AOCU/AOU average instruction 
speed is 200 ns and the smaller CCU and AOCU program 
memories have cycle times of 200 ns, no such speedup could 
be realized in these processors with instruction stacks, and 
none were provided. 

A fourth decision involved the output data bus from the 
parallel element to the Control Console. Although most ele¬ 
ment data are output from the AOUs to the AOCUs, occa¬ 
sionally the AUs contain data in registers destined for output 
to the ACU. To accomplish this transfer, the data could be 
stored back into element memory, retrieved by the AOU, 
output to the AOCU, and then transferred to the ACU. 
Alternatively, a separate AU/ACU data bus could be pro¬ 
vided. Since the first alternative was too slow and the second 
too expensive, it was decided to allow the AUs and AOUs 
to share the same output data bus, and logic to implement 
this was provided. Since the AUs output data infrequently 
relative to the AOUs, output operations are not significantly 
degraded. 

A final decision arose because the element CU, AU, and 
AOU share a common data memory, and since they can all 
operate simultaneously, conflicts inevitably occur and must 
be resolved. Since the element memory has a cycle time of 
100 ns, and the average instruction time is one microsecond 
for the AU and 200 ns for both the CU and the AOU, it 
would seem that either the CU or the AOU should have first 
priority. This priority, however, is problem-dependent so 
simulations were held to establish priorities for memory 
conflict resolutions. The simulations demonstrated that the 
priorities should be CU first, AOU second, and AU last. 
With this priority assignment, the AU program execution 
time was extended only about five percent over what it 
would be if the AU had exclusive memory access. 

Since memory access conflicts are rare (for typical BMD 
problems) and since the CU, AU, and AOU can operate 
simultaneously, average instruction rates for a single ele¬ 
ment can approach 11 million instructions per second. When 
all elements are operating, an ultimate rate of over a billion 
instructions per second can be achieved, but this rate is 
highly problem-dependent and would occur infrequently. 

PEPE APPLICATION IN BMD 

A simplified subset of the BMD data processing problem, 
as implemented on PEPE for its solution, will be described 
here to convey an idea of how PEPE is used in real-time 
control applications. Extensions to other similar data proc¬ 
essing problems should be obvious, or at least not difficult. 
The problem comprises the maintenance of data files in real 
time on all targets (typically several hundred) within view 
of the phased-array radar. This requires matching each radar 
return against all target files to determine which file should 
accept the return data, and placing the return data in that 
file (typically ten to a hundred instructions per return per 
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file). If no match is possible, the return is assumed to be 
from a new target and is used to initialize a new file. Then, 
the new return is used to update the target file (typically 
several thousand instructions per update). Finally, based on 
the updated file, a request for a subsequent radar pulse to 
gather new data on the target is generated (typically tens to 
several hundred instructions per update). The request is then 
scheduled on the radar time line based on a multiplicity of 
complex radar/environmental/situation constraints (typically 
several hundred instructions per file per request). The real¬ 
time constraint is that, for each return (several hundred to 
several thousand per second), a request for a subsequent 
radar pulse must be generated and delivered to the radar 
within a specified time interval. The time interval, generally 
called port-to-port time, is usually different for different 
functions and can be extremely critical for some of them. 

The problem is implemented on PEPE by assigning each 
individual target and its file to one element (other imple¬ 
mentations are possible but this is most straightforward). 
For simplification, initialization of files and starting the 
problem on the PEPE will be ignored; instead the course of 
the problem will be picked up in the middle of a BMD 
engagement and carried on from there. Several hundred 
target files occupy the ensemble memory, one target file to 
an element. Periodically, the files are updated in parallel by 
the ACU/AUs using appropriate mathematical routines. 
Generally, time enablement of the routines is used; the fre¬ 
quency of enablement is chosen high enough so that port- 
to-port timing restrictions are not exceeded. For any one 
enablement, not all files are updated; only those elements 
which have received new radar return data since the last 
update are activated and only they participate in the parallel 
updating. 

While the file updating is proceeding in the ACU/AUs, 
radar return data are streaming sequentially into the CCU 
data memory, either through the host or directly from the 
radar. As each return enters (a block of about ten 32-bit 
words describing target position, time of return, etc.) the 
CCU interrupts the ACU. The ACU then executes a pre¬ 
diction routine which predicts forward, to the time of the 
return, the positions of all targets on file. Then, the ACU 
constructs multidimensional windows around the positions. 
These operations are performed simultaneously for all tar¬ 
gets on file. The ACU is then released, and the CCU orders 
the transfer of all of the window parameters into the CU 
correlation registers. The new return data are broadcast to 
all CUs, and the target position is compared simultaneously 
to all windows. Where a comparison is successful, the return 
data are placed in the element where success was achieved, 
and are used to update the target file during the next AU 
update interval. When no successful comparisons are made, 
the target is assumed to be detected by the radar for the first 
time, and the return data are placed in an empty element to 
start a new target file. This sequence of events is continuous, 
and it occurs simultaneously and asynchronously with the 
updating operations taking place in the AUs. The sophisti¬ 
cated reader will note that the foregoing procedure is a gross 
simplification of the target correlation problem, no account 
being given for practical considerations such as crossing 


targets, ghosts, and redundant targets. However, such prob¬ 
lems are common to all computers and depend upon algo¬ 
rithms, not architecture, for their solution. As expected, the 
associative properties of PEPE make it an especially effi¬ 
cient implementer of algorithms designed to solve practical 
target correlation problems. 

As a final operation in each update occurring in the AUs, 
requests for subsequent radar transmit/receive pairs, one 
request per target, are generated. Request formats include 
time of pulse, type, beam direction cosines, range windows, 
special pulse characteristics, etc., depending upon target 
type, priority, and a variety of other considerations. Reso¬ 
lution of conflicting pulse requests, allocation of pulses to 
targets, and scheduling of requests in the form of pulses and 
receive windows on the radar time line, are tasks assigned 
to the AOCU/AOUs. Generally, the objective is to schedule 
each pulse so as to maximize the BMD system response at 
the current instant, subject to a large variety of constraints 
and demands. This requires that target files be searched for 
highest-priority requests, where priority is a function of 
many variables. These requests must then be matched 
against a set of dynamic constraints (generally stored in the 
Host) in order to build a block of radar orders. Obviously, 
the procedure calls for complex multidimensional file- 
searches and matching and comparison operations, and is 
extremely time and resource-consuming for conventional 
sequential computers with location-addressed memories. 
Since the AOCU/AOU conducts the file searches and makes 
the matches and comparisons associatively, the procedure 
is considerably simplified and speeded up. 

PEPE CONSTRUCTION 

PEPE construction is rather straightforward, and employs 
circuit devices, hardware, structure, and wiring which may 
be considered state-of-the-art for supercomputers. The Con¬ 
trol Console is contained in one cabinet 84" high, 82" wide, 
and 30" deep. Elements are contained in identical cabinets, 
36 elements to a cabinet. Eight element cabinets therefore 
comprise a full 288-element ensemble. 

A complete element is contained in six 16"x 18" printed 
circuit plug-in boards, two each for the AU and the AUO, 
and one each for the memory and the CU. All board wiring 
is contained in eight printed circuit layers, four for signals 
and four for voltages and ground. Signal layers are designed 
to maintain a constant 50-ohm impedance. The boards con¬ 
nect to cabinet back-plane wiring through four plug-in con¬ 
nectors per board, 100 pins per connector. All logic is im¬ 
plemented in MECL 10K Dual-in-Line (DIL) integrated 
circuit packages which plug into sockets on the boards, Each 
board has space for 300 DILs, but there are only about 1000 
DILs per element. 

The boards plug into the cabinets vertically in four rows, 
nine elements to a row. Each row has its own power supply 
which supplies 5.2 volts at 560 amperes and 2.0 volts at 540 
amperes to all of the elements in its row. The right-hand side 
of the cabinet contains these four power supplies arranged 
vertically, and the left hand side contains signal distribution 
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logic and circuitry. The four backplanes (one for each row) 
into which the elements are plugged are fabricated from 
three heavy laminated copper plates, which provide dc 
power distribution, signal ground plane, and power-supply 
bypass capacitance. Backplane interelement signal wiring is 
implemented entirely in 50-ohm subminiature coaxial cable. 
All of the nine elements in a row are connected to one signal 
distribution bus comprised of about 300 separate balanced- 
pair high-speed signal lines (PEPE uses a 10-mHz clock). 
The bus consists of flat belted cables which maintain con¬ 
stant impedance throughout the signal distribution system. 
The cables are connected to paddle boards which plug into 
the rear of the back plane to make connection to the ele¬ 
ments. The four busses in a cabinet are fanned out from 
signal distribution circuitry in the left end of the cabinet. 
The eight cabinets are all connected to the Control Console 
signal distribution system via balanced-pair, constant- 
impedance flat belted cables. 

Cooling is provided by a water-cooled chill plate located 
under the bottom row in the cabinet, and forced-air dual 
blowers located in a plenum beneath the cabinet. Total 
power dissipation is about 20 kW per element cabinet. 

The Control Console cabinet construction is similar to 
that of the element cabinet, except that the Control Console 
printed-circuit boards use only six printed circuit layers, 
plus some conventional wire-wrap signal interconnection. 

PEPE SUPPORT SOFTWARE 

A detailed discussion of PEPE software is beyond the 
scope of this paper, but a summary of the capabilities of the 
support software system developed for PEPE will be given 
here. The support software system comprises a real-time 
executive system, a PEPE operating system, an instruction- 
level hardware simulator, a procedure-oriented language and 
translation system, and a general utilities package. All of the 
foregoing operate under control of the CDC 7600 SCOPE 
2.0 Operating System. 

The real-time executive has two primary responsibilities. 
First, it schedules tasks for execution on the host and the 
various PEPE processors in accordance with requests gen¬ 
erated by other tasks, data enablements, time enablements, 
and system status. Second, it dispatches tasks in accordance 
with priorities and satisfied precedence relationships. The 
real-time executive resides and operates in both the CDC 
7600 Host and the PEPE ACU. 

The PEPE Operating System, which includes the real¬ 
time executive and supplies support service for it, has com¬ 
ponents which reside and operate in the CDC 7600 Host and 
all three control units. It includes mainly interrupt handlers 
and input/output handlers. Data buffers for Host/Control 
Console input/output operations, as well as storage space 
for the process control tables which support the real-time 
executive are also provided in the CDC 7600 small-core 
memory and the data memories in the PEPE control units. 

The instruction-level hardware simulator is a program 
which runs on the CDC 7600 and executes interpretively on 
real PEPE problem code to simulate the operation of the 


PEPE/Host configuration (except, of course, for run time). 
It was produced to support the development of PEPE system 
and problem software prior to hardware availability, but it 
has additional important uses. Since the initial PEPE hard¬ 
ware installation will comprise only 11 of the 288 elements, 
it will be used in combination with the hardware to run tests 
where 11 elements are insufficient. Also, since the entire 
PEPE Project is at present an experimental one, the simu¬ 
lator will be used to check proposed hardware modifications 
before they are implemented. 

PEPE is programmed in PFOR, a procedure-oriented, 
higher order superset of FORTRAN. PFOR also includes 
PAL, the PEPE Assembly Language. In summary, PFOR 
adds to FORTRAN additional statements which permit the 
declaration of parallel variables (those variables stored in the 
element memories), and which implement and exploit the 
parallel processing and associative access/grouping proper¬ 
ties of PEPE. The PFOR compiler is a set of compilers and 
assemblers, together with a linkage editor, which accepts a 
single PFOR source program and expands it into object 
programs and load modules for all of the PEPE processors. 
Current experience shows that PFOR is an easy language to 
use and PEPE is easy to program; FORTRAN programmers 
can learn to program PEPE in a week or so. PEPE programs 
are generally characterized by a structural simplicity which 
makes them easy to read, understand, debug, and modify. 
Of particular interest is the relative absence of complicated 
program loops and housekeeping operations on data loca¬ 
tions. 

The PEPE Utilities package is quite conventional; it com¬ 
prises the usual loaders, debug aids, and data recording 
programs. However, the actual implementation of the pack¬ 
age components is unique because of the PEPE parallel/ 
associative architecture. 


PEPE IN NON-BMD APPLICATIONS 

BMD data processing can be characterized as parallel 
computation on independent data sets; this characterization 
can also be applied to air defense, air traffic control, and 
several other applications. Such applications, when imple¬ 
mented on parallel architectures, do not benefit from a ca¬ 
pability for parallel interelement data transfer among the 
parallel elements; in fact, there are disadvantages, as dis¬ 
cussed previously in this paper, in providing such a transfer 
capability. However, there is a large class of problems which 
require operations on interdependent data sets, such as 
weather forecasting and other fluid dynamics problems re¬ 
quiring the numerical integration of nonlinear partial differ¬ 
ential equations. Such problems can be solved much faster 
when implemented on parallel architectures, but parallel 
interelement transfer capability seems, at first look, to be 
required for efficient execution. In this section of the paper, 
the operation of PEPE, which has no provision for parallel 
interelement data transfer, on such problems will be dis¬ 
cussed. 

In fluid dynamics problems, interest focuses on the time- 
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variant behavior of several dependent variables at a multi¬ 
plicity of fixed points in 3-dimensional space. This behavior 
is described (for a continuum) by a set of partial differential 
equations, generally nonlinear. The only practical proce¬ 
dures for integrating these equations are numerical ones, in 
which the region of interest is divided into a 3-dimensional 
grid, and the partial differential equations are replaced by 
their linear-difference approximations at each point in the 
grid. The resulting system of linear equations is then solved 
by any one of a number of methods developed for that 
purpose. Generally, these methods involve iteration and 
most of them fall within a broad class of operations called 
relaxation techniques. Computer architects do not need to 
know details of the specific algorithms used. It is sufficient 
to know that the relaxation procedure is started in a com¬ 
puter by assigning to each interior grid point estimated val¬ 
ues for the dependent variables. Further, variables at exte¬ 
rior grid points are assigned values fixed by boundary 
conditions or if not, estimated values. Then, for each grid 
point, the computer performs several hundred to several 
thousand mathematical operations (only the instruction mix 
and count are of interest to a computer architect) to “up¬ 
date” the values of the dependent variables at that point. 
These operations involve the variables at that point, and all 
the variables at the surrounding six points. After each grid 
point in the total volume of interest is thus treated (one 
relaxation), there are better estimates of the variables at all 
the grid points. The computer then repeats the whole pro¬ 
cedure for as many relaxations as are required to achieve 
convergence. The foregoing is a simplified and incomplete 
description of how a computer proceeds to solve problems 
in fluid dynamics, but it is sufficient for this paper. 

A sequential computer can operate on only one grid point 
at a time; therefore, run times can be extraordinarily long. 
Obviously, there is a considerable amount of parallelism in 
relaxation techniques of the type described above, so one 
should be able to do better with a parallel processor. The 
most direct approach would be to assign an element to each 
grid point, store the initial values of the variables at each 
grid point in its assigned element, and carry out all of the 
grid-point computations simultaneously. There are, how¬ 
ever, at least two problems which make this approach im¬ 
practical. First, there are requirements for fluid dynamics 
calculations in regions containing several hundred thousand 
grid points, and this is not a practical number of elements. 

Second, each element needs variables from other elements 
to perform its calculations, and moving these variables into 
the proper elements prior to calculations for each relaxation 
is a formidable engineering problem if a large number of 
elements are involved. 

The first problem can be solved by employing a mass storage 
device to store a complete grid image, and to have the 
parallel processor operate on one section of the grid image 
at a time. Thus, the variables at the grid points in a section 
of the grid image would be transferred to the parallel ele¬ 
ments, calculations would be performed on all of the grid 
points in that section, and the updated variables would be 
transferred back to that section in the grid image to replace 
the old values. Moreover, variables representing more than 


one grid point could be stored in an element, the number 
depending on the size of element memory. A complete 
sweep through the entire grid image would be made per 
relaxation, with the size of each section depending on the 
number of parallel elements and the number of grid points 
assigned per element. Sweeping through the grid image on 
mass storage and in effect replacing the image with a better 
estimate of it represents a lot of data transfer between the 
mass store and the parallel processor, but it cannot be 
avoided as long as there are not enough elements to contain 
the entire grid image, which will almost always be the case 
for the sizes of problems faced by fluid dynamicists. This 
data-transfer operation, incidentally, is independent of the 
second problem and is encountered in all parallel processors 
whether or not they have efficient facilities for transferring 
data among elements. The mass store data transfer problem 
must be handled by suitable choice of mass store and its 
interface to the parallel processor. 

The second problem can be handled in several different 
ways. First, provision can be made in the hardware for 
efficient interelement transfer, as was done in the ILLIAC 
IV. This complicates the hardware and forces a rigidly struc¬ 
tured implementation of the problem on the processor array, 
which in turn leads to reliability problems and perhaps top¬ 
ographical mapping problems with grids of unconventional 
geometry. 

Second, an external data manipulator, or permuter, can 
be attached to the parallel processor array to allow parallel 
transfer of data from any ordering of elements to any other 
ordering of elements. This is the approach taken by the 
STARAN flip network and Feng’s “Versatile Data Manip¬ 
ulator.” It requires a lot of extra hardware, but can handle 
any grid geometry. 

Third, interelement data transfer can be dispensed with 
entirely, as in the PEPE, so long as all of the data needed 
for one relaxation calculation is entered into the proper 
elements at the time a grid section is transferred from mass 
memory to the parallel processor array. This requires more 
storage per element, since each element must store not only 
the variables it is responsible for updating, but also all those 
additional variables needed to do the updating. This could 
mean a factor of up to six times the element storage required 
by the other two schemes. However, the hardware is simple, 
complicated grid geometries are no problem, and the parallel 
array could employ an associative data transfer scheme be¬ 
tween the mass storage and the array to counter the relia¬ 
bility problem of failed elements. This scheme would allow 
use of an essentially unmodified PEPE in fluid dynamic 
problems. 

Assume a global circulation problem, in which we place 
on the globe a grid point at every five degrees of latitude 
and longitude, and at six altitude levels. Unwrapped from 
the globe, this is a rectangular 72x36x6 grid. At each of 
these grid points, we are interested in the behavior of eight 
dependent variables. We will, therefore, be performing cal¬ 
culations at each grid point iteratively to continuously up¬ 
date the values of the variables. To perform one update of 
the eight variables at one grid point requires that we have 
available the previous eight values at that point and the 
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current 48 values from the six neighboring points (north, 
south, east, west, above, below), and that we must perform 
2000 arithmetic operations for each grid point. One complete 
relaxation requires an update of all 36x72x6=15,552 grid 
points. All of the foregoing numbers are realistic, being 
derived from the NCAR Global Circulation Model. 

To solve the above problem on PEPE, we store the entire 
grid image on mass storage. We assume the mass storage is 
connected, via simple interface, to the CCU and the AOCU. 
Alternatively, it could be connected to the host via a stand¬ 
ard interface. Now our general approach is to compute on 
a section of the grid at a time, to always store the required 
neighboring grid points in the same element as the point 
being updated so that interelement data transfer is never 
needed, and to simultaneously compute, input, and output 
on three different grid sections. Specifically, for each grid- 
section relaxation we will assign to each element responsi¬ 
bility for updating the six points in one column. That means 
we store in each element one column and the four neigh¬ 
boring columns, or 5x6x8=240 words. Now we have all 
the data needed to update the six points in the center col¬ 
umn. Since PEPE will be simultaneously computing on one 
grid section, inputting from mass store on the next, and 
outputting to mass store the updated values of the previous 
section, we need 3x240=720 words of storage per element. 

To update the center column requires 2000 instructions 
per grid point, or 6x2000=12000 per column. Since the 
PEPE AU computes at a rate of one MIP, this takes 12 ms. 
Now, how many columns can be updated simultaneously, 
or stated another way, how many elements can be kept 
busy? The answer will give the performance that can be 
obtained from PEPE, or the number of MIPS that can be 
applied to the problem. 

As soon as one set of columns is updated, computation 
should start on the next set of columns, to keep the AUs 
busy continuously. That means the loading of the next five 
columns in the next grid section, and the unloading of the 
previous five columns should be complete in 12 ms. Un¬ 
loading (center column only) of 48 words/element at 2.4 /as/ 
word= 115.2 /ts/element (2.4 /is/word is the AOU to AOCU 
transfer time; we assume that AOCU to mass memory time 
can keep up with this). Therefore 12000/115.2=107 elements 
can be unloaded during the 12 ms compute time. 

Loading is a little more complicated. Five columns per 
element must be loaded in 12 ms, but five elements can be 
loaded simultaneously, provided we can steer the columns 
to the right elements. If we can do this perfectly, then 
loading time will be the same as if we only had to load one 
column per element. Assume we cannot do it perfectly, so 
that we have to load the equivalent of two columns per 
element. Then, loading takes 96 words/element at 1.2 /as/ 
word, so 107 elements can be loaded during the 12 ms 
compute time (1.2 /As/word is the CCU to CU transfer time). 

All of the foregoing means we can keep 107 elements con- 
continuously busy , giving a computational rate of 107 MIPS 
on the global circulation problem. This is better than any 
other existing machine can achieve, especially when one 
considers that the MIPS required for I/O are overlapped 
with the compute MIPS and do not subtract from them. In 


other machines, of equivalent MIP rating, the I/O MIPS 
(and/or interelement transfer MIPS) must be subtracted, 
which effectively lowers computational rates. 

There are several advantages to the foregoing implemen¬ 
tation. The formulation is independent of grid size and ge¬ 
ometry; these can be changed easily. The implementation is 
straightforward and should be easy to program. Data man¬ 
agement should be simple, especially when compared to that 
required for sequential and vector machines. The use of 
associative input to the elements should permit element fail¬ 
ures without program abort, so throughput per shift should 
be higher than can be obtained from less reliable machines 
of equivalent MIP rating. Finally, the more sophisticated 
and complex the updating algorithms are (which is the 
trend), the better the performance. For instance, 4000 in¬ 
structions (rather than 2000) would result in keeping 214 
elements continuously busy, or 214 MIPS. 

ARCHITECTURAL ENHANCEMENT TO THE PEPE 

The major architectural improvement that can be made in 
the present PEPE design is simplification of the signal dis¬ 
tribution system. Such a modification would result in a ca¬ 
pability for incorporating recent technological advances, pri¬ 
marily the use of available bit-slice microprocessor chips, 
into the element design. It would also result in an ability of 
several subsets of the PEPE ensemble to be executing pro¬ 
grams at the same time, instead of only one as in the present 
design. The signal distribution system of PEPE is also the 
limiting physical factor involved in increasing the number of 
elements in a parallel processor. Studies show that the PEPE 
signal distribution system (drivers, receivers, logic trans¬ 
mission lines, connectors) can be simplified about two or¬ 
ders of magnitude. The price paid is some additional com¬ 
plication in the elements. To understand this, consider the 
original concept of PEPE. A single control unit, containing 
instruction memory, and instruction fetching, interpretation, 
and decoding circuitry drives a large number of execution 
units, each with its own memory. This concept of multiple 
execution units and only one control unit saves a great deal 
of hardware in a parallel processor, because the control unit 
in a computer generally contains a significant amount of 
complicated hardware. However, the interface between the 
control unit and the execution unit is the most complicated 
interface in a computer, and it is this interface which, under 
the original PEPE concept, must be carried by the signal 
distribution system to all elements. At the time of the orig¬ 
inal PEPE concept, engineering tradeoffs favored the single 
control unit and the complicated distribution system. 

Tradeoffs would produce different results now. With thou¬ 
sands of elements, one probably cannot tolerate a compli¬ 
cated signal distribution system. Moreover, control unit cir¬ 
cuitry is much cheaper; in fact, microprocessors contain 
both instruction decoder and execution unit on a single chip. 
The interface between the instruction decoder and the exe¬ 
cution unit is not even available outside the chip. 

So, one can put both the instruction decoder and the 
execution unit in the element much easier and cheaper than 
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Figure 5—New PEPE scheme 


was possible a few years ago, by making extensive use of 
LSI in the elements (possibly even off-the-shelf micropro¬ 
cessor chips). Then, the PEPE control unit would contain 
only a program memory and instruction fetching circuitry. 
This control unit broadcasts only the opcode to the ele¬ 
ments, rather than decoded opcode signals. Fewer than ten 
signal paths are required to broadcast opcodes, rather than 
a hundred or so, as is the case when the decoded opcode 
must be broadcast. Moreover, the signal levels on the signal 
distribution system must change only once every instruc¬ 
tion, rather than once every clock pulse. Thus, for the same 
instruction rate, the switching signals on the signal distri¬ 
bution system can be about ten times slower. These two 
factors, fewer signal paths and slower switching speeds, will 
permit great simplification in the signal distribution system. 

Figure 5 shows the scheme. Only the details of the AU 
are shown, although the CU and AOU can be handled iden¬ 
tically. The AU contains a microprocessor chip, which con¬ 
tains both instruction decoder and execution unit. In oper¬ 
ation, the simplified ACU in the control console fetches an 


instruction, and places the address on the element address 
bus and the opcode on the opcode bus. The opcode first 
enters the data pins on the microprocessor chip, then the 
data from element memory is entered into the same pins. 
The instruction is decoded and executed on the chip and the 
result is returned to element memory. This sequence is con¬ 
ventional for almost all microprocessors. A further refine¬ 
ment can be made by storing special subroutines, such as 
trigonometric or exponentials, in ROM in the AU. Thus, the 
control console can include these functions within its opcode 
set and slow down the signal distribution system switching 
speed even more, while actually increasing the PEPE in¬ 
struction rate. 

Other useful operations can be done with a PEPE confi¬ 
gured as described above. Since the ACU executes on the 
average about ten microinstructions for every opcode, more 
than one subset of AUs (same statement goes for CUs and 
AOUs) can be active at the same time, which is not possible 
with the present PEPE design. All that is needed is for the 
opcode bus to carry an activity number with it; only that set 
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of elements holding that activity number would execute that 
opcode. Then, while the selected subset of elements is ex¬ 
ecuting the instruction, another instruction could be broad¬ 
cast on the same bus to a different subset of elements. 

This type of operation (it could be viewed as the logical 
equivalent of two or more ensembles) would keep more 
elements busy at the same time. Even further, the ensemble 
elements could store complete subroutines in their program 
memories, so that the control units need only broadcast 
subroutine calls rather than instructions. This would de¬ 
crease signal-distribution system traffic even more, and 
would allow further exploitation of the ability of the subsets 
of the ensemble to execute several operations simultane¬ 
ously. 
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PEPE—A user’s viewpoint; a powerful 
real time adjunct 


by M. P. MARIANI and E. J. HENRY 

TRW—Defense and Space Systems Group 
Redondo Beach, California 


INTRODUCTION 

The Parallel Element Processing Ensemble (PEPE) offers a 
cost-effective means of obtaining significant improvements 
in data driven real time systems. The inherent parallelism 
and programmability of the PEPE architecture make it an 
extremely effective growth option for data driven systems 
developed around a large-scale centralized Host. With the 
PEPE architecture, incremental system growth with mini¬ 
mum software impact can become a reality. 

The purpose of this paper was to investigate the imple¬ 
mentation of PEPE in real time data driven systems. For the 
purpose of an example, we have chosen the Ballistic Missile 
Defense (BMD) problem—a real time highly data driven 
system in its own right. The BMD problem desires a data 
processing system architecture which is not only highly re¬ 
silient to the threat but also provides for cost-effective 
growth to accommodate changing technology. With the 
PEPE architecture, the system growth can be achieved 
through the addition of PEPE elements—the system de¬ 
signer can incrementally purchase only those parallel proc¬ 
essing elements needed. 


THE BMD DATA PROCESSING SYSTEM 

ARCHITECTURE 

The assumed data processing subsystem (DPS) used in 
this study is responsible for controlling the radar and inter¬ 
ceptor subsystems to insure that a sufficient number of de¬ 
fended missiles survives an offensive ICBM attack. To ac¬ 
complish this objective the DP subsystem performs twelve 
basic functions (Figure 1). These functions are categorized 
as either module level or unit level. 

The module level functions are oriented toward opera¬ 
tional control of multiple Defense Units operating asyn¬ 
chronously. The unit level functions perform the detection, 
discrimination and intercept of the threatening objects within 
the Unit’s operational sector. Each Defense Unit contains 
a CDC 7700 data processor, a single, phased array radar and 
a complement of interceptors. BMD research is currently 
focused on only the Defense Unit level of the system, and 
the first eight basic functions. 

The eight unit level functions constitute the application 


programs (AP) operating in each Defense Unit’s CDC 7700. 
The AP contains an array of independent tasks which are 
either data or time enabled, and scheduled and dispatched 
by the real time operating system. Of the array of tasks, a 
subset constitutes the bulk of the real time resource load on 
the CDC 7700. 

Normally, each of the applications tasks consists of mul¬ 
tiple subroutine modules which, for testability and growth, 
are usually limited in size. Within each task is a task control 
routine which controls the execution sequence of the sub¬ 
routines within the task, performs all global data base READ 
operations prior to task execution, and performs all global 
data base WRITE operations at the termination of the task 
execution. This access approach reduces the real time data 
management overhead. 

The array of tasks can range in size from less than 2500 
executable instructions* to well over 40000 executable in¬ 
structions for each instance of data processed. In order to 
prevent any single task from monopolizing the system, a 
maximum task execution time can be imposed. This design 
constraint limits the number of data instances which can be 
processed per task execution. As a result, there are large 
variations from task-to-task, in the number of instances 
which could be processed per enablement. These variations 
when combined with variations in task execution priorities 
can result in queue buildups. In high threat load situations 
the queue buildups of some of the more data sensitive tasks 
can experience increases in execution wait times. 

Under severe loads, system saturation will be character¬ 
ized by queue overflows and corresponding increases in 
critical response time. The parallelism of PEPE provides the 
potential for handling such high load conditions by reducing 
the data processing system’s sensitivity to the threat and 
increasing the system threshold of saturation. 

PEPE ARCHITECTURAL OVERVIEW 

The PEPE is a 32-bit super-computer of the SIMD and 
MIMD 1 class. The PEPE 2 (Figure 2) can contain up to 288 
parallel processing elements. Each element is itself a mul¬ 
tiprocessor containing three independently operating pro- 

* Average Machine Language Instructions (MLI) derived from simulations of 
the applications process over a full engagement scenario. 
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cessors—an input correlation unit (CU) and an arithmetic 
output unit (AOU) operating at 5 MIPS*, and an arithmetic 
unit (AU) operating at 1 MIPS. Each processor in the mul¬ 
tiprocessor element is under the control of a separate control 
unit which broadcasts decoded parallel instructions simul¬ 
taneously to all 288 of the processing elements. The control 
units are themselves processors capable of executing se¬ 
quential instructions while simultaneously issuing parallel 
instructions to the PEPE elements. The PEPE architecture, 
therefore, permits the concurrent execution of as many as 
six instruction streams; three serial instruction streams driv¬ 
ing the control unit computations and three parallel instruc¬ 
tion streams driving each of the three processors in the 
parallel elements. 

In an operational environment in which PEPE is used as 
♦Based on typical BMD computational instruction mixes. 


an adjunct processor to a large high speed Host, such as the 
CDC 7700, the PEPE will execute only highly data parallel 
computations. The PEPE will operate in a dedicated manner 
and will contain its own application load modules and op¬ 
erating system. The PEPE-Host communications will be 
used to only support process control and global data shared 
between separate processes executing on the Host and 
PEPE. 

PEPE IMPLEMENTATION—OBJECTIVE 

The parallel architecture of PEPE, coupled with its inher¬ 
ent throughput potential offers an available low cost method 
for responding to the advances in threat technology. An 
implementation of PEPE which does not accompany a rede¬ 
sign of the software results in an inefficient use of the PEPE 


LEVEL 

NAME 

DEFINITION 

Unit Level 

1. RRA/RS 

Radar Return Assimilation 
and Radar Scheduling 


2. ODD 

Object Detection and 
Designation 


3. 0T 

Object Track and Prediction 


4. IPP 

Interceptor Planning and 
Prelaunch 


5. IC 

Interceptor Guidance 
and Control 


6. URM 

Unit (Radar and DPS) 

Resource Management 


7. DMC 

Defense Module Inter¬ 
communication 


8. 0D 

Object Discrimination 

Module Level 

9. MBM 

Module Battle Management 


10, MSA 

Module Status Assessment 


11. ESA 

Environment Status 

Assessment 


12. CMP 

Command Message Processing 


Figure 1—Basic BMD functions 











PEPE 


995 


EXTERNAL 

I/O 


EXTERNAL 

I/O 


EXTERNAL 

I/O 


PROGRAM 

MEMORY 


PROGRAM 

MEMORY 


PROGRAM 

MEMORY 


CORRELATION 
CONTROL 
UNIT (CCU) 


ARITHMETIC 
CONTROL 
UNIT (ACU) 


ARITHMETIC 
OUTPUT 
CONTROL 
UNIT (AOCU) 


INTERCOMPUTER 
COMMUNICATION 
LINK (ICL) 



PARALLEL ELEMENT NO. 288 


Figure 2—PEPE architecture overview 


resources. However, the optimal allocation of a PEPE ar¬ 
chitecture to the dual CPU CDC 7700 architecture could 
involve substantial breakage.* The initial objective of the 

*Breakage as related to the current DP system architecture (both software 
and hardware) was considered to include: 
a. New Architecture (SW or HW) components, 
b Modification of existing Architecture components, 
c. Disposing of existing Architecture components. 


investigation was then to consider alternatives for low-cost 
(minimal breakage) implementations of PEPE. Therefore, 
any offload design requiring either a significant software 
redesign and/or a significant redesign of the PEPE-Host- 
Radar hardware interface was not considered. Similarly the 
optimization of the offloaded code was considered less crit¬ 
ical than maintaining the original task software. 

For each offload design considered both the application 
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task and operating system software were analyzed for break¬ 
age. Results of potential offload designs indicated that the 
application software breakage was highly sensitive to the 
number of application tasks included in the offload, while 
the indirect software breakage of the CDC 7700 operating 
system is essentially insensitive to the number of application 
tasks offloaded. 

PEPE IMPLEMENTATION 

Under the minimum breakage guidelines, there were two 
fundamental PEPE-Host architectures which were consid¬ 
ered (Figure 3). One architecture (Figure 3B) employed 
PEPE as a radar interface preprocessor to the Host. As a 
preprocessor the PEPE would perform the radar interface 
function; correlation of radar returns and scheduling of radar 
pulses. However, designs of the radar interface software for 
centralized computers would execute extremely inefficiently 
on the PEPE, and the necessary software modification to 
improve the efficiency would be significant. The second 
architecture (Figure 3C) employed PEPE as an offload pro¬ 
cessor to the CDC 7700 in its current configuration. As an 
offload processor the PEPE would be dedicated to perform¬ 
ing such highly data parallel operations as verification of 
detection returns, track initiate and track and discrimination 
processing. 

Analysis of conventional applications programs was used 
as a basis for developing the offload options and selection 
of the preferred offload configuration. The analysis provided 
performance data which expressed resource requirements 
for: task computation (TASK), operating system support 
directly related to task execution (TOSC) (e.g., data base 
access), and operating system support indirectly related to 
task execution (e.g., external communication). All analysis 
was conducted within the high load periods of the engage¬ 
ment scenarios investigated. 


PEPE IMPLEMENTATION—DESIGN APPROACH 

Offload candidate identification 

In an attempt to achieve the most cost-effective offload, 
the applications tasks were evaluated with respect to their 
current CDC 7700 resource requirements and their PEPE 
offload efficiency. Three aspects of the task resource re¬ 
quirements were studied, 

(1) Task Execution (TASK): the resources (msec) required 
to execute the task instructions only. 

(2) OS Chargeable (TOSC): the overhead directly asso¬ 
ciated with task execution (e.g., OS service requests: 
task load from secondary store, file read and write, 
task termination, etc.) 

(3) Other OS (TOSO): the overhead not directly associ¬ 
ated with task execution; not occurring during task 
execution (e.g., external I/O, interrupt handling, 
scheduling and dispatching). 


The resource breakdown was used to provide an indication 
of each task’s overhead sensitivity to the execution of in¬ 
struction (TASK), handling of global data (TOSC), and ex¬ 
ternal communication and polling loop priority (TOSO). 
Only those tasks which exhibited high resource require¬ 
ments were considered as candidates for offload to PEPE. 
In addition, the general sensitivity of the tasks to the threat 
traffic was investigated by studying each in further detail 
during high load periods. 

Those tasks which were rated as high resource users were 
then examined for inherent parallelism (both data and logic), 
as a basis for assessing the efficiency (breakage) with which 
a candidate task could be offloaded to the PEPE. Data 
parallelism was related directly to the task input data rates. 
With the minimum breakage guideline, algorithmic parallel¬ 
ism was not considered; instead, logic parallelism was eval¬ 
uated at the subroutine level using the functional flow block 
diagrams. Tasks with a high degree of conditional branching 
along equal probability paths (Pb -40-60 percent) would re¬ 
sult in an inefficient use of PEPE resources. The lock-step 
architecture of PEPE can cause degraded performance in 
key processing tasks directly proportional to the number of 
branches in the offloaded processing logic. 

A collection of six basic performance parameters was 
identified as providing the basis for assessing PEPE offload 
potential; they are, 

(1) Total resource requirement 

(2) Total number of instances processed 

(3) Task resource distribution: TASK, TOSC, TOSO 

(4) Total number of Task executions 

(5) Peak data rates 

(6) Task polling loop wait 

Based on these performance parameters a set of five (5) 
performance criteria was established for paring down the 
application tasks to preferred offload candidates. These cri¬ 
teria were, 

(1) high frequency of task execution 

(2) high number of instances processed 

(3) high instruction execution resources 

(4) low data handling overhead 

(5) high data queue buildup 

In addition, the task structure was evaluated to identify 
those tasks with 

(1) high probability logic paths (Pb n £:90 percent) 

(2) low offload instruction count with high frequency ex¬ 
ecution. 

Using the above criteria the applications tasks were evalu¬ 
ated for their resource requirement. The results indicated 
that a small subset of the applications tasks accounted for 
over 90 percent of the total system resource requirements. 

Analysis during a prolonged engagement interval provided 
performance profiles for the principal resource users. How¬ 
ever, the analysis did little to indicate the sensitivity of the 
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Figure 3—BMD terminal defense architectures 
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tasks to threat traffic. Any offload configuration was re¬ 
stricted by the critically short processing response thresh¬ 
olds and limited offload storage available on PEPE. As a 
result, there was a need to develop criteria with which to 
assess the relative offload potential of each candidate task. 
The initial criterion was the resource offload percentage 
available due to each task. Tasks whose individual Host 
resource requirements failed to exceed at least five percent 
of the total applications program, were not considered as 
potential offload candidates. This test singled out five tasks 
as having the independent resource potential to justify 
offload. 

The throughput capability of the PEPE relative to the 
Host, and the inherent parallelism of the PEPE architecture 
dictated two data related requirements. First, in order to 
exploit PEPE’s parallelism large data loads were desired by 
the tasks for offload. Second, the cost to process a single 
instance should preferably be small when taking into ac¬ 
count the potential expansion in processing times due to the 
relative PEPE-CDC clock cycles (100ns-27.5ns). 

The first criterion strongly favored three of the tasks, with 
two of the tasks being marginal. One task did not meet the 
throughput requirement although the task’s resource re¬ 
quirement is among the highest. 

The second criterion of single instance processing costs 
strongly favors four tasks showing a high throughput re¬ 
quirement, with one task (interceptor control) exhibiting 
extremely high single instance processing costs. In addition, 
the stringent response requirement of interceptor control 
also makes it a poor selection for offload to PEPE. 

A third performance consideration is the potential PEPE- 
Host communication overhead accompanying each task off¬ 
load to PEPE. Large data communication overhead has neg¬ 
ative impacts on both the PEPE-Host interface bandwidth 
requirements, and the responsiveness of the offloaded proc¬ 
essing. Since the TOSC resource component is primarily 
influenced by data management costs associated with the 
global data, the TOSC component is a strong indicator of 
the potential PEPE-Host interface overhead. The ratio of 
TOSC to TASK resource distribution was then used to eval¬ 
uate the inherent resource offload efficiency available with 
each offload candidate. The ratio ranged from approximately 
.25 to 1.14 and was found to remain fairly constant regard¬ 
less of system loading. 

Another consideration for evaluating task offload potential 
was the frequency of task execution. The execution rate of 
tasks when compared with their associated data rates pro¬ 
vided another indication as to the efficiency with which 
tasks will execute on the PEPE. Each task execution is 
accompanied by a basic overhead cost (TOSC) consisting of 
loading the task module from large core memory (LCM) and 
the READ/WRITE of global data which support the task 
execution. This basic overhead is virtually insensitive to the 
number of data instances processed during a single task 
execution. If an execution time restriction were not imposed 
on the process designer, it would be much more efficient to 
have each task totally deplete the data queue on each exe¬ 
cution. The parallel processing architecture of PEPE pro¬ 
vides a means of processing a much larger number of data 


instances per task execution while still maintaining an exe¬ 
cution time design restriction. 

Offload design—General approach 

Minimization of the impact of any offload design required 
the development of a technique for restructuring the current 
application software to support offload to PEPE. The tech¬ 
nique which we developed was based on partitioning the 
applications tasks along their current subroutine structure. 
In addition, the table driven design for the current operating 
system allowed the offload to be implemented with no im¬ 
pact to the OS software. 

The partitioning approach to PEPE offload involved the 
segmentation of each offload task into three components 
(Figure 4) operating as three tasks. Two new data files were 
defined which serve as data enablement queues for the sec¬ 
ond and third segments. The first and third segments of the 
task would continue to reside in the Host computer. These 
segments provide all access to the global data base, and 
interface with the remainder of the application process. The 
second segment would reside on the PEPE. One objective 
of the partitioning was to maintain the total cost of executing 
the two segments residing on the Host =£the original cost of 
the task prior to offload. 

In general the segmentation will be performed across 
loosely coupled routine interfaces in the task logic, and 
produce a segmentation in which the largest segment of the 
task is placed on the PEPE. 

The initial segment will continue to execute in the Host 
as a data enabled task. The primary purpose of this Host 
segment will be to format and write into the newly defined 
PEPE input queue all data necessary to support the second 
task segment residing in the PEPE. The data will typically 
include the data instances, such as radar returns, and dy¬ 
namic data which is needed to support the execution of the 
PEPE segment. 

The second segment would be offloaded to the PEPE and 
allocated to either the CCU, ACU, or AOCU. The PEPE 
will contain the complete load module of the second segment 
loaded into the program memories of the CCU, ACU, or 
AOCU. Any fixed point arithmetic or logical operations of 
the offloaded segment, such as unpacking/packing opera¬ 
tions, will generally be performed by the CCU or AOCU. 
The ACU will perform all floating point computations. In 



Figure 4 —Task offload segmentation technique 
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addition, new PEPE software must be developed for the 
CCU and AOCU to support data manipulation within the 
PEPE, such as loading the PEPE elements prior to parallel 
computations and retrieving the processed data from the 
elements for retransmission back to the Host following com¬ 
putations. In addition, for data management operations such 
as large file searches/correlations, new software must be 
incorporated into the offloaded segment which exploits the 
PEPE’s associative capabilities. The PEPE Fortran (PFOR) 
possesses instructions specifically designed to support as¬ 
sociative search operations within the PEPE elements. Typ¬ 
ically operations such as object redundancy checks and nu¬ 
clear interference prediction which could be extremely 
costly when implemented on the Host, will be executed with 
only a couple of associative search instructions on PEPE. 

Alternative offload designs 

One of the major applications tasks consistently displayed 
performance characteristics indicative of efficient PEPE 
offload. That task performed the object track function. The 
keys to the offload desirability of the track task were four 
operational characteristics, 

• high task execution frequency with highest execution 
rates occurring during periods of highest threat traffic. 

• high task throughput requirement with highest data 
loads (75 percent) occurring during periods of highest 
threat traffic. 

• high CDC resource requirement with largest load (80 
percent) occurring during periods of highest threat 
traffic. 

• low task execution overhead (30 percent) throughout 
the operational period. 

Because of the task’s offload desirability it was used to form 
the foundation of two alternative offload designs. 

In addition, the task has a moderate per instance execu¬ 
tion cost, a highly favorable characteristic for tasks belong¬ 
ing to short response threads. 

Using the track task as a basis, two multi-task offload 
designs were developed. The first design was directed at 
significantly reducing the total Host resources requirement. 
The second design was directed at substantially improving 
the responsiveness of key processing threads in high threat 
traffic situations. The first design incorporated the detection 
and track initiate and discrimination processing tasks with 
the track task into an offload module for the PEPE. This 
design provided the largest offload benefit in the early por¬ 
tion of the engagement. In addition, the detection and track 
initiate tasks are on processing threads with less stringent 
response requirements than the track thread which reduces 
the impact of resource contention. The second offload in¬ 
corporated the interceptor control task with the track task 
into an offload module for the PEPE. The interceptor control 
task was a high resource user with an expensive per instance 
execution cost. In addition, the interceptor control task was 
on the most stringent response processing thread in the 
applications program. 


A comparison of the two integrated offload designs indi¬ 
cated the first design to be far more preferable. The offload 
of a sizable portion of the Host resources, also had a 
significant indirect effect on improving the responsiveness 
of the threads remaining on the Host. The offload to PEPE 
of tasks within high response processing threads had less 
effect on improving system responsiveness than the first 
offload design. This was attributed to the fact that, (1) less 
absolute resources were offloaded from the Host to PEPE, 
and (2) the offload of tasks with equal response priority 
created resource contention on the PEPE. An important 
point of both offload designs was the ability of PEPE to 
substantially improve the performance and predictability of 
both system throughput and responsiveness. 

PEPE IMPLEMENTATION—ANALYSIS 
Offload design — S/W portability analysis 

Parallel Fortran (PFOR) 

Parallel Fortran 3 is a high level, procedural extension to 
USA Standard Fortran and provides a powerful and efficient 
set of language extensions for the associative parallel proc¬ 
essing capabilities of PEPE. Through the use of the PFOR 
statements, the user has the capability of simultaneous ex¬ 
ecution of instructions in all n PEPE elements or a selected 
subset of the n elements. The user can nest the levels of 
element activity through PFOR. 

In order to identify specific variables which are allocated 
to PEPE element memory, PFOR has parallel type decla¬ 
ration statements. 4 A set of parallel “system” functions to 
handle type conversion, square root, and special mathemat¬ 
ical functions on parallel arguments and a set of powerful 
statements are included in PFOR for the subsetting of the 
current set of active elements. This set includes statements 
of the form WHERE, CONVERGE, parallel IF, and parallel 
DO. In addition, a statement is provided for moving data 
between element memories. 


CDC 7700 S/W conversion 

In order to determine the amount of code breakage and 
time required to convert existing code in FORTRAN to 
PFOR (Parallel FORTRAN), typical code representative of 
the object track task was converted. The routines were 
selected as being strongly indicative of the types of code 
which were contained in the PEPE segments of the offload 
design. 

Some of the code was highly logic oriented and designed 
to control the hi-fidelity Kalman filter for the track task. 
The other code contained mostly arithmetic operations and 
little branching. 

There are two options which can be considered when 
placing the selected subroutines and their associated data 
base on PEPE. Option one permits only one object main¬ 
tained per PEPE element and requires a larger set of PEPE 
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CPC 7700/RN 


PEPE/PFOR (OPTION 1) 


SUBROUTINE XYZ 


SUBROUTINE XYZ, ACU 


COMMON/D3TOOOO/A3TOO (132) 
INTEGER ZDBQN, . . . 
EQUIVALENCE (A3TOO(1), ZDBQN) 

IF (Z3D0F .EQ.O) GO TO 10 

Z3BPF = 1 


PAR C0MM0N/D3T0000/ZDBQN, . . . 
PAR INTEGER ZDBNQ 

WHERE (Z3D0F.NE.O) 10 
Z3BPF = 1 


10 CONTINUE 


10 CONTINUE 


IF (Z3D0F.EQ.O) GO TO 70 

ZNC (1, Z30DV) = ZNC (1, Z30DV) + 
FL0AT (ZNCNL (Z3BQV)) 

GO TO 80 

70 ZNC (1, Z30DV) = ZNC (1, Z30DV) + 
FL0AT (ZNCNL (Z3BQV)) + 1. 

80 CONTINUE 


WHERE (Z3D0F.NE.O) 70 

70 ZNC (1) = tNC (1) + PFL0AT (ZNCNL(Z3BQV)) 

WHERE (Z3D0F.EQ.O) 80 

ZNC (1) = ZNC (1) + PFL0AT (ZNCNL(Z3B0V)) + 1. 
80 CONTINUE 


Figure 5—Relative code comparison 


elements. In converting the code for this option, the user 
must change the doubly subscripted variables currently used 
throughout the software (e.g., object state estimate) into 
singly subscripted variables. The second subscript associ¬ 
ated with the instance number (e.g., object identification) 
would be implied by mapping each instance into a unique 
PEPE element. This option results in significant code mod¬ 
ification, however, the code conversion would be simple, 
straightforward and could be automated. 

Option two would permit more than one object maintained 
per PEPE element and can significantly reduce the size of 
the PEPE element array required. This option requires dou¬ 
bly subscripted variables to identify the current instance 
being processed in an active element. This option would 
result in much less code breakage than option one but would 
require much larger data storage per element. Once the user 
develops some familiarity with PFOR a set of existing rou¬ 
tines can be transformed into PFOR in relatively short time. 
As an illustration, Figure 5 provides an example of a logical 
segment of the subroutine and the FORTRAN code changes 
required to convert it to PFOR. In practically all cases of 
code conversion, the code which required modification was 
directly associated with data management and manipulation. 
The actual scientific computations written in FORTRAN 
were directly transportable to the PEPE. For the three sub¬ 
routines that were selected, approximately 76 percent of the 
executable statements would require modification for option 
one and 4 percent for option two. 

In the offload design, the current CDC 7700 data base 
design is not effective when directly installed on PEPE. In 
each offload design the subroutines executing on the PEPE 
require only small segments of the data contained in current 


common blocks. To conserve element memory we elected 
to restructure the common blocks within the PEPE. The 
restructuring involves eliminating unused variables and re¬ 
defining the nonexecutable statements which define these 
variables. Since the current set of declarative statements 
(i.e., INTEGER, REAL, EQUIVALENCE) have to be con¬ 
verted to PFOR (i.e., PAR INTEGER, PAR REAL) there 
is total breakage for these nonexecutable statements. 

It is felt that other applications which are suited to the 
type of parallel processing offered by PEPE could just as 
easily be converted to PFOR. In some cases the code con¬ 
version could even be automated. Applications which would 
require movement of data between elements may require a 
longer time to convert to efficient PFOR, but once the user 
becomes familiar with PFOR, the code conversion is 
straightforward. 


Offload design—performance impact 

Analysis indicates that the first offload design exhibits a 
significant overall potential for reducing the load on the Host 
computer. Over the complete engagement the offload design 
provides a net reduction in the average Host resource re¬ 
quirement of the applications program by over 25 percent. 
This is accomplished by a 54 percent reduction in the re¬ 
source requirements originally required by the offloaded 
tasks. The offload design provides the most significant re¬ 
ductions during the major peak load periods of the engage¬ 
ment; specifically early during initial acquisition of the threat 
and later during track and discrimination prior to intercept 
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commit of the threat. The early load is caused by high Host 
resource requirements imposed by the track initiate loads. 
The later Host load is caused by high Host resource require¬ 
ments imposed by the object track and discrimination proc¬ 
essing. The offload to the PEPE provides a significant re¬ 
duction in the Host loads during both the early (—28 percent) 
and later (—23 percent) portions of the engagement. 

An analysis of the PEPE under this offload design indi¬ 
cated that both storage and utilization requirements were 
well within available limits. The program storage require¬ 
ments for each of the three control units were all less than 
35 percent of available storage. Data storage requirements 
were higher and reached 84 percent of that available in the 
CCU, and 65 percent of that available in the PEPE element 
memories. Evaluation of the PEPE resource requirement 
during the engagement indicated that none of the control 
units exceeded 80 percent utilization. The average utilization 
of these units was substantially less, never exceeding 60 
percent (ACU), indicating resources are available for addi¬ 
tional applications tasks to be offloaded. 

The size of the Host resource offload is directly related 
to the traffic load on the system and as a result varies during 
the engagement. A major benefit of the PEPE offload design 
is the reduction in sensitivity of the PEPE-Host system 
resource requirement to increases and unpredictable pertur¬ 
bations in data (threat) traffic patterns. 

The impact of resource offloads on the system respon¬ 
siveness was also investigated. In the original Host environ¬ 
ment there were two principal components of system re¬ 
sponse, the first component being that of execution wait 
times, which includes the time from enablement to dispatch. 
The second component is the actual execution time once 
dispatched. In high load periods of resource contention, the 
task wait times become a major component of the overall 
system responsiveness for the Host alone system. With the 
offload to the PEPE, a third resource component must be 
considered, that of PEPE-Host communication. In the off¬ 
load design, the contention for resources is reduced, and the 
execution wait times are decreased. In addition, the exe¬ 
cution wait times tend to be more stable with the PEPE 
adjunct than with the Host alone. 

When combining the effects of reduced task execution 
wait time, increased task throughput and the addition of the 
communication delays, the results indicate that the PEPE- 
Host system provides improved system responsiveness. 
This responsiveness is substantially improved during periods 
of high load conditions. 

One of the major offload issues addressed the impact that 
the PEPE adjunct, and its associated PEPE-Host commu¬ 
nication, had on the Host Large Core Memory (LCM) uti¬ 
lization overhead. Analysis indicated that the LCM over¬ 
head actually decreased with the offload. Although the Host 
LCM overhead necessary to support real time input/output 
(RTIO) increased with the addition of the PEPE, the reduc¬ 
tion in the overhead associated with management of some 
of the larger data files and load modules, now resident in 
the PEPE, more than compensated for the RTIO overhead 
increase. 


SUMMARY 

THE PEPE provides a highly cost-effective method of im¬ 
proving the data handling capacity of a real time system 
developed around a single large-scale Host. Analysis of con¬ 
ventional terminal defense systems for BMD indicates that 
the addition of a PEPE adjunct can provide a means to 
substantially offload the Host computer. As a result of the 
Host offload, an improvement can be achieved in the ca¬ 
pacity and predictability of both the system’s throughput 
and responsiveness. 

In addition, the implementation of PEPE can be accom¬ 
plished with only a minimum breakage to software devel¬ 
oped for the Host. In most cases the software breakage is 
repetitive but not complex and can be automated without 
significant cost. For the specific offload design investigated, 
the PEPE storage and throughput requirements were well 
within limitations and PEPE had the resources to accom¬ 
modate additional offloads. Based on the current design the 
apparent threshold in PEPE offload will be the availability 
of data storage. Analysis indicated that through a redesign 
of some of the larger common areas the data storage re¬ 
quirements on the PEPE can be significantly reduced. 

Further analysis of the offload design indicated that the 
PEPE adjunct provided several direct performance improve¬ 
ments for the data processing system, 

(1) increased data throughput for tasks offloaded to the 
PEPE, 

(2) reduced execution delays for entire application pro¬ 
gram, 

(3) reduced data management overhead, 

(4) reduced system sensitivity to data traffic, 

(5) improved predictability of the system’s real time re¬ 
source demand. 

It is felt that these performance improvements can be 
achieved as well with non-BMD applications which possess 
the parallelism which is suited for the PEPE. Once the PEPE 
user becomes familiar with the operation and the computing 
power of the PEPE, suitable applications can be mapped 
onto PEPE in a simple and straightforward manner. 

CONCLUSIONS 

Results of this PEPE application case study have general 
applicability to most data driven, real time systems. The 
inherent parallelism and programmability of the PEPE make 
it an extremely attractive mechanism for achieving signifi¬ 
cant improvements in a centralized systems data handling 
capability. 

The implementation of PEPE as a Host adjunct allows the 
offload of resources to the PEPE. This offload can produce 
a substantial improvment in the data processing system’s 
throughput and response. The improvement is effected in 
both the system’s capacity and performance predictability 
to data loading. 
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Special computer architectures for pattern recognition 
and image processing—An overview 

by K. S. FU 

Purdue University 
West Lafayette, Indiana 


INTRODUCTION 

It has been recognized for a long time that a conventional 
sequential processor is inefficient for operations on pictorial 
data where relatively simple operations need to be per¬ 
formed on a large number of data elements (pixels). 

Though many parallel processing architectures for picture 
processing have been proposed in the past, very few have 
actually been implemented due to the costs involved. With 
LSI technology, it is becoming possible to realize parallel 
architectures at a modest cost. 

In the following we review some of the proposed archi¬ 
tectures for pattern recognition and image processing. 

PREVIOUS WORK AND COMMENTS 

Previous work in the field of special purpose computer 
architecture for image processing basically falls into two 
categories: bit-plane processing and distributed processing. 
The bit-plane processing approach performs the arithmetic 
computation on the image points which are stored in Boo¬ 
lean bit-plane. For example, if the image has eight grey 
levels, then the image points are stored in three Boolean 
planes. The bit-plane processing approach tends to have a 
large number of processors which perform Boolean opera¬ 
tions. The distributed computing approach utilizes proces¬ 
sors which have powerful computation capability. These 
processors are designated by microprogram or hardware to 
execute certain specific tasks. Thus, this configuration forms 
a distributed computing architecture. In this section we will 
briefly illustrate each special purpose computer, concen¬ 
trating on the special features of each computer for image 
processing and pattern recognition. 

Unger 1 realized as early as 1958 that a sequential proces¬ 
sor is inefficient for problems involving spatial relations and 
proposed an array computer (Figure 1). The system is made 
up of central control (Master Control) controlling a large 
rectangular array of identical modules interconnected as 
shown. The master control performs all instruction fetches 
and decoding and broadcasts the decoded instruction to all 
the modules in the array. 

Each module consists of a 1 bit accumulator and a small 
number of memory elements (a,b,c . . .). Inputs to each 


module are from the M.C. and the accumulators of the four 
neighbors (above, below, left, right). Each module can per¬ 
form the following basic operations: 

1. LOGICAL OR (AND) 

The accumulator is ORed (ANDed) with specified 
memory register (a,b,c . . .) and the inputs from the 
specified neighbors. The results are either stored in 
the accumulator or one of the memory registers. 

2. LOGICAL NOT 

3. LOAD (STORE) 

4. SHIFT (ROTATE) LEFT (RIGHT,UP,DOWN) 

The entire image is shifted (rotated) in the direction 
specified. 

5. LINK/EXPAND H (V,UP,DN) 

These operators are used to determine the connectivity in 
the specified direction. Thus, for example, it is possible to 
determine all the cells in a given field which are connected 
to a specified cell through a chain of horizontally linked ones 
(Figure 2). Using these commands, many interesting image 
processing operations can be performed. But the actual pro¬ 
gramming of algorithms is nontrivial. 

Unger’s architecture has influenced that of many later 
researchers. Kruse’s PPM, the CLIP array, as well as the 
ILLIAC III computer were inspired by it. 


1LLIAC 111 

The Illinois Pattern Recognition Computer, ILLIAC III, 2 
was designed for automatic scanning and analysis of massive 
amounts of relatively homogeneous visual data, in particu¬ 
lar, bubble chamber negatives. The computation is per¬ 
formed by three units, the pattern articulation unit (PAU), 
the taxicrinic unit, and the arithmetic unit. The pattern ar¬ 
ticulation unit performs local preprocessing on the digitized 
raster, such as track thinning, gap filling, line element rec¬ 
ognition, etc. The logic design of the digital computer has 
been optimized for the idealization of the input image to a 
line drawing. Nodes representing end points, bends, points 
of intersection are labeled in parallel by appropriate pro¬ 
grams. The abstract graph describing the interconnection of 
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Figure 1—Module interconnection pattern for Unger’s machine 


labeled nodes is then extracted as a list structure, which 
comprises the normal output of the processing element (sta¬ 
lactite). The pattern articulation unit consists of an iterative 
array of 1024 identical stalactites (32x32). A schematic dia¬ 
gram of the stalactite is shown in Figure 3. The iterative 
array of the stalactite can be connected in, either rectan¬ 
gularly or hexagonally, at the programmer’s option. Each 
stalactite can accept an input from any of the eight neigh¬ 
boring stalactites, SO, SI, ... , and S7 shown in Figure 3 
(rectangular) or six neighboring stalactites (hexagonal) in the 
plane and from itself, M shown in Figure 3. The input signals 
are logically “OR”ed, optionally complemented, and stored 
in one (or more) of the nine planes. Communication with the 
supplementary planes in the core buffer is through the “M” 
plane, which serves as the buffer register of this memory. 
For output, the output of any selected set of planes can be 
logically “AND”ed and passed on to neighboring stalactites. 
With a special signal, the stalactite allows an input signal to 
pass through it directly, without interim storage. It is this 
feature which allows path-building within the machine. The 
set of 32x32 stalactites processes the pixels of a 32x32 pixel 
image window simultaneously. The whole image is to be 
processed by one image window (32x32) after another se¬ 
quentially by the set of 32x32 stalactites. The taxicrinic unit 
assembles the graphs, which are outputs of the pattern ar¬ 
ticulation unit, into coherent list structures and categorizes 
these graphs to complete the visual recognition function. 
The arithmetic unit is designed for executing mathematical 
analysis, such as stereoreconstruction, statistical summari¬ 
zation, etc. involved in processing pictorial data. The block 
diagram of ILLIAC III is illustrated in Figure 4. 



Figure 2—Link/expand operator 


The generic stalactite contains 10 one-bit registers (flip 
flops). Since asynchronous logic is employed, it is required 
that no flip flop be both read from and written into during 
the same instruction to avoid races. 

The pattern articulation instruction code is of 40 bits 
length, made up of 4 groups. 

Flow group 

C G Cl F B U D N2 N1 NO 
1 10 
C, Cl are used to control the outgoing and incoming signal 
complementation respectively. 

G=1 indicates that at least one input gate is opened. 

F=1 indicates a flash through operation, i.e., the input 
signal is transferred directly to the output bypassing the 
stalactite. 

B=1 indicates a bubbling operation. 

In bubbling the values of the nearest neighbors are read. 
Using a sort, the bubble register (flip flop 1,2,. . . ,8) 15 
ordered such that zeros are at the top and the ones at the 
bottom. The number of ones is counted and the cell is 
marked if it exceeds a threshold. This operation is used 
extensively in line thinning and other preprocessing opera¬ 
tions. U specifies bubble up, D specifies bubble down, N2, 
Nl, NO specify the number of iterations needed for bubbling. 

FF Read control group 

M01234567 8 

1 10 
This field is used to select the FF’s to be read out. 

Input gate control group 

C01234567 8 

1 10 

C=0 All inputs signals enabled C = 1 Only specified signals 
enabled 

FF Load control group 

M01234567 8 

1 10 
This field specifies the FF’s to be loaded. 

Mark command 

This command is of the form MARK(Z). It sets the M 
register of the cell at coordinate Z=(X,Y). 

LIST Command 

List serially compiles a list of all coordinates Z which 
have a zero in the M FF. Coordinate Z of the next point 
found is transferred to the LIST register automatically on 
storage of the previous contents of the register. 

The use of the above two instructions is illustrated below. 
A sparsely populated image is complemented and loaded 
into M reg of the PAU array. The black points in the image 
correspond to a 0 in the M plane. A first black point is 
identified by the LIST command. The coordinate obtained 
is used to mark the corresponding M FF (and thus erase the 
point). Now the black points connected to this point are in 
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Figure 4 —Block diagram of Illinois pattern recognition computer ILLIAC 

III 
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turn listed using Flash through. This technique builds a 
paired list of nodes, i.e., the branches of the associated 
graph. 

One special feature of ILLIAC III for image processing 
is the use of an auxiliary memory. The auxiliary memory is 
used for the processing of images frame to frame, and for 
retaining intermediate partial results. The necessity of stor¬ 
ing intermediate results of the iterative array stems directly 
from the manner of executing homogeneous logic transfor¬ 
mations in the processor. The complexity of the iterative 
array can be greatly simplified if various digital filtering 
operations can be performed serially storing the intermediate 
results (32x32=1024 bit words). For example, local track 
segment orientation can be designated as being predomi¬ 
nantly horizontal, vertical, left-diagonal, or right-diagonal in 
four serial tests, whereas simultaneous identification requires 
approximately four times as much hardware. 2 Unfortu¬ 
nately, ILLIAC III has never been completely built since 
1967. 

The ILLIAC III represents a pioneering effort in designing 
a special processor for a specific image processing task. The 
processing capability is, however, limited due to the tech¬ 
nology available at the time. The complexity of the system 
can be imagined from the fact that each stalactite occupies 
one circuit board, and 1024 boards make up the PAU array. 
Recent advances in LSI technology make building of such 
arrays economically viable as demonstrated in such systems 
as the CLIP array described later. By using more complex 
logic in the modules, it is possible to realize truly general 
image processing capability. 

CLIP Series 

Digital Parallel Processors (DPP’s), 3-8 using cellular 
logic, are real-time machines in which the action resulting 
from a program statement is simultaneous on all the points 
of the array. The action may be symmetrical or directional 


and the tessellations of various types, the most common 
being the square and the hexagonal ones. An example of 
this kind of machine is the parallel cellular logic image pro¬ 
cessor, CLIP 3. 6,7 The CLIP 3 is comprised of an array of 
192 cells arranged in a block of dimensions 16 cells (verti¬ 
cally) by 12 cells (horizontally). The array interconnection 
pattern can be either square or hexagonal. The required 
architecture is determined by one bit in each instruction 
word and is therefore under the control of the programmer. 
The block diagram of the cell logic of CLIP 3 is shown in 
Figure 5. The Boolean processor can perform, under pro¬ 
gram control, two independent Boolean functions from its 
two inputs A and P. D is one output and can be regarded as 
the processed pattern bit corresponding to the cell. The 
other output, N, fans out to the adjacent cells. The N inputs 
from neighboring cells (G x , G 2 , . . . , and G 8 ) are summed 
(2) and compared with a threshold (f). The programmer is 
allowed, for each PROCESS instruction, to select several 
inputs to be put into the summing unit and he can also 
choose the threshold value. The summed and thresholded 
output, T, is then “OR”ed with the second pattern input to 
form the input, P, to the Boolean processor. D outputs are 
addressed into any one of 16 single bits of store; buffers A 
and B are loaded from addressed location in the same store. 
The CLIP 3 was actually built. 

One drawback of both the ILLIAC III and CLIP 3 is that 
the “edges” caused by moving image windows, (in ILLIAC 
III 32x32 windows and 16x 12 windows in CLIP 3)) are not 
taken care of by the computer. The computing power of the 
CLIP 3 is obviously limited because of the fixed small num¬ 
ber of (Boolean operator) cells. (16x12=192 cells). 

In order to process larger images, CLIP 3 has been inter¬ 
faced to a television camera through the hardwired scanning 
unit. This is called a hybrid CLIP 3 array 8 which is shown 
schematically in Figure 6. This unit scans the 192 cell CLIP 
3 array across the 96 by 96 cell data-field and provides edge 
stores to handle the propagation signals which cross between 
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Figure 5—CLIP 3 cell logic 
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Figure 6—Schematical diagram of hybrid CLIP 3 system 


adjacent sectors. The complete system is interfaced to a 
PDP 11/10 computer which serves to extend the available 
data and instruction storage and also provides program ed¬ 
iting and assembling facilities. 

CLIP 4 8 is the Large Scale Integrated circuit (LSI) version 


of CLIP 3 with some small changes in the cell design. The 
CLIP 4 uses N-MOS (N type Metal Oxide Silicon) LSI to 
incorporate eight processor cells in a four by two block onto 
one chip. The block diagram of CLIP 4 cell logic is shown 
in Figure 7. The “D” store has been increased from the 8 
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Figure 7—CLIP 4 cell logic diagram 
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bits of CLIP 3 to 32 bits which allow the processing of 32 
grey levels. The interconnecting threshold gate has been 
replaced by an “OR” gate, and a few extra gates and an 
additional buffer store have been included to provide auto¬ 
matic carry-in arithmetic operations. 

The experimental CLIP 4 system consists of 16x12 iden¬ 
tical cells. The array interconnection pattern can be either 
square or hexagonal with six neighbors. The desired archi¬ 
tecture is programmed by a bit in the instruction. The pro¬ 
cessor in the CLIP 4 is a Boolean processor capable of 
implementing any Boolean function of inputs A,P. Thus, 

D=b2(P,A), N=b3(P,A) 

The actual functions b2,b3 are determined by the control 
lines cl, c2, . . ., c4 and c5, c6, . . . , c8 respectively. (All 
16 possible Boolean functions are implemented for D and 
N). 

The B register is primarily intended for comparing two 
images. D is a 32 bit random access storage in each module. 
The input gating is used to enable/disable inputs from neigh¬ 
boring modules. This permits selective propagation of the 
interconnection signal in desired directions within the array. 
Thus, 

T=G1.I1+G2.I2+. . .+G8.18+G9). Carry in 

Carryin is a carry bit used to perform bit serial arithmetic. 

The normal mode of operation of the array is as follows. 
The input image is digitized by a flying spot scanner and 
loaded into the A register of the array modules. Once the A 
registers are loaded with the desired image we can perform 
operations on it. The results may be stored in the D memory. 
A CRT is used to display the output. 

Instruction set 

The array is controlled by a sequence of instruction words 
stored in external memory. Instructions which do not in¬ 
volve signal propagation through the entire array are exe¬ 
cuted in 1 microsecond. Delays for operations involving 
propagation depend on the array size as well as the image 
being processed. 

There are basically three types of instructions recognized 
by the assembler: 

1. LOAD 

LD (A addr), (B addr), (D addr) 

The instruction loads A and B registers from the locations 
specified in the D memory. The D address is used by the 
subsequent PROCESS instructions as the destination ad¬ 
dress for the operations. 

2. PROCESS 

PR (Direction list) b3, b2, E S 

The ‘Direction list’ specifies which neighbor inputs are to 
be enabled (Gl, . . . , G8). b3 is the Boolean function for 
the interconnection output, b2 is the Boolean function for 


the image output, which is stored in the D memory location 
specified by the previous LOAD instruction. E is a binary 
number (0 or 1) specifying the inputs at the array edges and 
S indicates whether square or hexagonal interconnection 
pattern is desired. 

3. BRANCH 

This is a conventional program branch instruction. 

Analysis of CLIP operation 

Each cell in the CLIP array may be partitioned into three 
parts: 

1. Next state logic 

2. Propagation Logic 

3. Memory 

The interconnect signal propagation logic is combinational 
and there is no delay between elements. Thus, during each 
clock period, signals propagate through the entire array, the 
propagation depending on the function b3 (set by c5, . . . , 
c8). 

For simplicity in the following analysis, we assume that 
register B=0, c9=0. The operation is best studied in terms 
of interconnect signal propagation as below. 

1. ZERO ORDER 

Here D=0 or 1 independent of interconnection signal. 

Thus this is a trivial case. 

2. FIRST ORDER 

In this case D=A or D=NOT.A independent of N. 

3. SECOND ORDER 

D=b2(A,T), N=b3(A) 

Note that T is the input from the neighbors. Also the 
interconnect signal N depends only on the module state A 
and not on the interconnection signal input T. Thus the 
output of each module depends only on the current state of 
the module and that of its immediate neighbors (determined 
by Gl, . . . G8). Hence the maximum delay is only 2+t 
where t is the delay per module. 

4. THIRD ORDER 

These functions result when N depends both on the input 
T as well as the state of A. All stable functions in this 
category are third order functions. Since interconnect signal 
propagation occurs through the entire array the execution 
of these instructions is slow. These are the most interesting 
functions that may be executed on the CLIP array. 

5. FOURTH ORDER 

These correspond to those operations for which the output 
is unpredictable due to logical races within the array. An 
example is the operation 

N=A.T 

Two consecutive 1 cells can result in a latch up with unpre¬ 
dictable output. 
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Comparing the performance of CLIP 3 and CLIP 4, the 
processing speed of the CLIP 4 cell is slower than that of 
CLIP 3 by at least a factor of five. A single operation which 
required a pair of instructions (LOAD and PROCESS) in 
CLIP 3, taking 2 /Asec, is expected to require about 10 /as 
in CLIP 4. Propagation from cell to cell took 0.1 /as in CLIP 
3 and may take 1 /as in CLIP 4. 8 However, the processing 
speed of the overall system of the CLIP 4 is improved by 
the larger array of processors of CLIP 4 (96x96 cells). 


PPM and PICAP 

The Parallel Picture Processing Machine (PPM) 9 is a spe¬ 
cial processor which is connected to and controlled by a 
conventional computer. The block diagram of PPM is shown 
in Figure 8(a). The PPM consists of the following principal 
parts: the processing unit, a set of nine general purpose 
picture registers, and a control unit. The picture registers 
which are shown in Figure 8(b) comprise nine shift registers 



Figure 8(a)—Block diagram of parallel picture processing machine 


Operand 

address 


Result 

address 



rata out frees 
register 8^ 

Cata in to 
register 9j 


Figure 8(b)—Picture registers 


control lines 



0q (from neighbour¬ 
hood ) 


(froT picture 
registers) 


Figure 8(c)—Neighborhood matching logic 



Figure 8(d)—Block diagram of PICAP computer system 
















1010 


National Computer Conference, 1978 


capable of storing a picture. The main parts of the processing 
unit are the neighborhood matching logic (NML), shown in 
Figure 8(c), the line buffers (LB’s) and neighborhood count¬ 
ing register (NCR) and coordinate register (COR). The de¬ 
mand for LB’s stems from the fact that when a picture is 
stored in a picture register, only one picture point at a time 
is accessible. To be able to perform the local neighborhood 
operations, all nine neighborhood units must be accessed 
simultaneously. Therefore, two LB’s are added. Note that 
the speed improvement is accomplished mainly at the ex¬ 
pense of NML. The control unit is conventional in all re¬ 
spects except for the writable memory where the templates 
of an operation to be performed are stored. The parallel 
picture processor in Reference 9 has been linked with an 
ordinary minicomputer as part of the computation facilities 
of the PICAP picture processing laboratory 10 at the Univer¬ 
sity of Linkoping, Sweden. The block diagram of the system 
is shown in Figure 8(d). The function of the minicomputer 
is twofold: (1) it controls the other devices and (2) it per¬ 
forms all nonpictorial data processing. PICAP which is a 
modified version of PPM is dedicated to the several tasks of 
the picture processing, such as fingerprint coding and ma¬ 
laria parasite detection. An important modification of PICAP 
has been the addition of a set of registers for the collection 
of measurements. The most important task of the picture 
processor is to produce measurements of application-de¬ 
pendent features. That is, to reduce the often enormous 
amount of information in an image to a set of feature meas¬ 
urements that can be handled by a conventional computer. 

Each module in PPM is a synchronous sequential circuit 
with inputs from the 8 neighbors and the controller. All 
modules change states simultaneously. 

STATE TRANSITION FUNCTION—Q(ij)+=d(V,Q(ij), 
Q(ij-fl), . . . , Q(i+lJ)) 

OUTPUT FUNCTION—U(ij)=Q(ij) [Moore model] 

Since a Moore model is assumed, signal propagation 
through many layers of combinational logic is avoided and 
results in faster execution. 

The system possesses gray scale capability since Q(iJ) 
can be many bits wide (say n) and can represent 2**n gray 
levels. This permits arithmetic operations on pictures. (Dig¬ 
ital filtering, thresholding, image enhancement, etc.) In 
order to avoid long control words to fully specify the desired 
state transition function (extremely large number of state 
transition functions are possible, but few are useful in prac¬ 
tice) a template matching scheme is proposed. Each module 
is capable of storing a list of templates, against which the 
current state vector is compared, and if a match is found a 
state change to the specified state occurs. 

It is also possible to define arithmetic operations on pic¬ 
tures. For example: 

0 1 0 
f(P)= 1 -4 1 

0 1 0 

can be used as the template to find the discrete Laplacian 


of a picture. Here Q(i, j)+ = -4*Q(i, j)+Q(i-l, j)+ 
Q(i, j-l)+Q(i+l, j)+Q(i, j + 1). Operation on two or more 
pictures is possible by using multiple arrays with interarray 
communication. Conceptually this system can perform vir¬ 
tually any local operation. Unfortunately, hardware costs 
prohibit a fully parallel implementation of all features. 

CDC flexible processor 

Using the approach of distributed computation for design¬ 
ing a special purpose computer for image processing, the 
Control Data Corporation designed the Flexible Processor. 11 
The Flexible Processor was developed for a large digital 
change detection system for concurrent processing of four 
channels of side-looking radar imagery. The Flexible Pro¬ 
cessor is a microprogrammable processor and uses random 
access memory up to 1024 words of 48 bits each for micro¬ 
program control. One special feature of the Flexible Pro¬ 
cessor is its data transmission which is shown in Figure 9. 
Four separate balanced party-line transmission channels are 
available. Each 32-bit channel can initiate and receive reg¬ 
ister-buffered transfers. A scanning system allows several 
Flexible Processors on a party-line to communicate with 
each other and with peripheral devices. For output trans¬ 
mission, a 32-bit output channel is provided which interfaces 
on balanced lines. A dual internal data bus system is used 



Figure 9—Block diagram of flexible processor by Control Data Corporation 
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to match data transfer speed to the speed of arithmetic logic. 
The arithmetic unit consists of an arithmetic logic unit, hard¬ 
ware network for conditional microinstruction execution, 
array hardware multiplier, specialized logic for square root 
and divide, and hardware priority interrupt mechanism. 


TOSPICS 

The TOSPICS, developed by Toshiba Company called 
Toshiba Image Processing System TOSPICS, 12 ’ 13 is an in¬ 
teractive image processing system which is a disk-based 
system and each operation is performed by the command 
input through a teletypewriter. The software system for the 
image processing system consists of the permanent and non¬ 
permanent resident system programs, the picture proces¬ 
sors, the block common area, and a package of image proc¬ 
essing programs. The system diagram of TOSPICS is shown 
in Figure 10. The special features of TOSPICS for image 
processing are that the image memory is commonly used in 
order to reduce the amount of data transmission and the 
parallel picture processor is constructed to perform certain 



Figure 10—Block diagram of TOSPICS computer 



Figure 11—Functional block diagram of ILLIAC IV 


programs at high speed. The parallel picture processor per¬ 
forms the program of spatial filtering at a speed of 1 pixel/1 
\x sec. The I/O devices of TOSPICS include a unique high 
precision flying spot scanner by a Double Deflection Tube 
(DDT) with 4096 x 4096 resolvable points and 32,000x32,000 
addressable points. 

Comparing the systems of Flexible Processors and TOS¬ 
PICS, the basic ideas of parallel processing for image proc¬ 
essing by the distributed computing approach are the same 
for these two systems. The difference between these two 
computer systems are the ways of utilizing processing ele¬ 
ments. The TOSPICS uses the minicomputer, TOSBAC- 
40C, as an image processor and a parallel picture processor 
is attached to it. The parallel picture processor is used for 
certain programs, such as spatial filtering, affine transfor¬ 
mations, histogram calculation, and density conversions. 12 
The computer system of Flexible Processors uses the Flex¬ 
ible Processors as the processing elements. Each Flexible 
Processor is assigned for one task and several Flexible Pro¬ 
cessors are organized as the computer system. 


ILLIAC IV 

The ILLIAC IV 14 * 15 is a large array processor designed in 
the mid 60’s to solve complex scientific problems involving 
arrays (matrices). The system was designed to have 256 
processing elements (PE’s) divided into four quadrants to 
achieve a total throughput rate of 1000 MIPS. However, due 
to cost overrun, only one quadrant has been built so far. 
The ILLIAC IY is patterned after the SOLOMON computer 
and employs one master control unit (CU) for instruction 
fetching, decoding and scalar operations. All processors in 
the array execute the same command (Figure 11). 

Each processing element (PE) may be in one of the two 
modes: active/inactive, thus permitting some modules to be 
selectively disabled. Each PE has access to its own 2048 
words of 64 bit memory called a PE memory (PEM). Each 
PE is a sophisticated arithmetic and logic unit capable of 
performing a wide range of operations. The PE’s have been 
specially designed to perform floating point operations on 
64 and 32 bit words very fast. The CU is also capable of 
executing its own instructions in parallel with the PE’s. The 
CU has access to all the PEM’s and fetches instructions 
from them. A broadcast facility permits common operands 
to be fetched by the C and ‘broadcast’ to all the PE’s. Each 
PE has communication paths to nearest left and right neigh- 
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bors as well as to PE’s at distance 8 on each side. The 
ILLIAC IV is under control of a Burroughs B-6500 proces¬ 
sor which handles all I/O and operating system tasks in a 
conventional manner. 

The ILLIAC IV is a powerful array processor for opera¬ 
tions on matrices, vectors, etc. However, since most in¬ 
structions on this system are geared toward 64 or 32 bit 
floating point operations while most image processing is 
done on integer arrays of 8 or 12 bit length, there is consid¬ 
erable mismatch and results in many problems. If only one 
pixel value is stored per ILLIAC word, most of the memory 
is wasted whereas packing results in execution time ineffi¬ 
ciencies. (Eight-bit arithmetic has been recently incorpo¬ 
rated in its hardware. 25 ) 

The ILLIAC IV is organized as a linear array rather than 
as a two-dimensional array. The only routing signals being 
to the nearest neighbors much of the spatial connectivity 
relations have to be established by software. 

STARAN 

The STARAN 16 computer is a parallel processing com¬ 
puter built by Goodyear Aerospace Company in 1972. 
STARAN was not designed as a special purpose computer 
for image processing. But this computer was utilized for the 
task of resampling in the field of image processing in 
1977. 17,18 The STARAN parallel processor is a single instruc¬ 
tion stream multiple data stream (SIMD) processor. 19 The 
previously reviewed computers, such as, ILLIAC III, CIP 4, 
PICAP, and TOSPICS are also SIMD processors. The Flex¬ 
ible Processor is a SISD processor. The single most impor¬ 
tant element of STARAN is the associative array, which 
provides content addressability and parallel instruction ex¬ 
ecution capabilities. Most STARAN computing is done 
within a word of associative array memory. An associative 
array word is normally divided into fields of varying lengths 
by the programmer to suit the requirements of specific pro¬ 
grams. The values of these fields then can be added, sub¬ 
tracted, multiplied, and divided within the word. The 
STARAN can perform the same operations as a sequential 
processor but with the added capability of performing these 
operations simultaneously on literally thousands of words in 
the associative processor arrays. A basic STARAN config¬ 
uration contains an associative array. However, up to 32 
associative arrays can be included in a single STARAN 
system. The block diagram of the STARAN computer is 
shown in Figure 12. The sequential controller provides off¬ 
line capabilities for assembling and debugging STARAN 
programs and control of STARAN error processing, diag¬ 
nostic, and maintenance programs. Buffered I/O is available 
for tying different types of peripherals into the STARAN 
control memory. Also BI/O can be used to transfer blocks 
of data and/or programs between the STARAN control 
memory and host memory. The external function (EXF) 
logic facilitates coordination between the different elements 
of STARAN for special functions and simplifies housekeep¬ 
ing, maintenance, and test functions. By issuing external 
function codes to the EXF logic, elements of STARAN can 
control and interrogate the status of other elements. In gen- 



Figure 12—Block diagram of STARAN computer 


eral, the STARAN computer is powerful compared with the 
Flexible Processor and TOSPICS. However, only one kind 
of image processing task (resampling) has been carried out 
on this computer. Therefore, the effectiveness and efficiency 
of STARAN for all types of image processing and pattern 
recognition is still an open question. 

Each STARAN array consists of 256 processing elements 
connected to a (256x256) bit multidimensional access mem¬ 
ory. The array is controlled by a conventional sequential 
processor. The array memory and routing system permit 
memory access in many different modes. For example, it is 
possible to access in either column mode or row mode. 
Hence, we can process either all bits within a word in par¬ 
allel or the same field of all desired words in parallel. Shifting 
capability has also been provided for various data manipu¬ 
lations such as the familiar butterfly computation for eval¬ 
uating Fast Fourier transforms. 

It is possible to use an associative processor as a parallel 
processor to operate on pixels simultaneously. For this the 
associative nature of the processor is not utilized but rather 
the parallel processing capability of the associative proces¬ 
sor array. On the other hand it is possible to utilize the 
content addressibility of associative processors to simplify 
certain problems in image processing. Several illustrative 
examples can be found in Reference 22. 

REMARKS 

Consider the attributes of the bit-plane processing ap¬ 
proach and the distributed computing approach. The bit- 
plane approach mostly uses Boolean operators as proces¬ 
sors. Boolean operators work only on binary images which 
are not common in the real world. One way to get around 
this is to use several binary picture planes to represent the 
grey scale values of picture points. But, the complex soft¬ 
ware and additional memory requirement cause another 
problem and this problem limits the processing power of the 
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processor. The distributed computing approach appears to 
be more feasible considering the present state-of-the-art with 
respect to both software and hardware. Almost all the spe¬ 
cial computers discussed are only designed and tested in 
terms of signal-level processing 21 (such as image filtering, 
thresholding, thinning, and edge detection). There has been 
very little attention at the symbolic level, such as syntactic 
and semantic processing. A major drawback of previous 
computers is that the system’s processors are not reconfi- 
gurable. The vast varieties of sensor types, applications, and 
image processing techniques, require that the image proc¬ 
essing system (especially the parallel processor) be recon- 
figurable. Therefore, a generalized computer architecture 
which is reconfigurable under software control appears to 
be appropriate for the many applications of image process¬ 
ing. One of such a system has recently been proposed. 23 In 
the proposed computer architecture parallelism with the task 
is exploited by a multi-microprocessor parallel processor. In 
the meantime the operations of the sequential arithmetic 
processor are pipelined with the parallel processor under 
programmer control in certain tasks which can be decom¬ 
posed into pipelined processing. Therefore, parallelism and 
pipelining are exploited at the same time in the proposed 
computer architecture. 

In some special applications, the use of an auxiliary mem¬ 
ory for image processing has been implemented in conven¬ 
tional computer facilities for image processing and pattern 
recognition research. One example is described in Reference 
24. The computing facility is organized around a DEC PDP 
11/45 digital computer with 32K core memory and 96K fast 
secondary memory, two disk drives, a magnetic tape drive, 
two cassette tape drives, a line printer, and a CRT monitor. 
The computer is not a special purpose computer, but the 
auxiliary memory is a special feature for image processing. 
The auxiliary memory was developed for image processing 
mainly because of the limited addressing range of the 16-bit 
minicomputer. Therefore, the limited addressable memory 
is difficult to use for implementing large programs. A mem¬ 
ory controller has been built which can access up to 2 32 
bytes of memory and which now controls 64K 16-bit words 
of core memory. The controller is interfaced to the PDP 11/ 
45 unibus through essentially three 16-bit registers. Two 
registers form an effective address of 32 bits, and the third 
register contains the data to be written into the memory or 
the data read from memory. Once the address is loaded on 
the two registers, data can be written into or read from 
memory in less than one microsecond. Since large arrays of 
data can be addressed without I/O to the disks, execution 
times improve. Also, computer programs become simpler 
since no sophisticated disk-core swapping software is 
needed. An experiment on rib identification in chest x-ray 
images was programmed once without using the auxiliary 
memory and once using this memory. On the average, the 
execution time improved a factor of five. Other software 
shows comparable execution time. 
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Experience with a picture processor in pattern recognition 
processing* 

by BJORN KRUSE 

Linkoping University 
Sweden 


INTRODUCTION 

In recent years much effort has been devoted to the study 
of picture processing algorithms and pictorial pattern rec¬ 
ognition. The efforts have resulted in both theoretical un¬ 
derstanding of and practical approaches to some of the basic 
problems involved. 

Common to the areas of picture processing is the great 
need for processing power. One picture may contain as 
much as several million picture elements (pixels). A scene 
viewed through a common television camera for example, 
represents approximately two megabits of data renewed 25 
times per second. To produce or analyze data at such rates 
is completely beyond the capacity of the general purpose 
computer. Even if the speed is increased by a factor of ten, 
which may be anticipated in the future, the GPC will fall 
short when faced with real-time applications. The problems 
stem from the fact that pictures represent a new type of 
data, orders of magnitude richer in information than that for 
which the GPCs architecture was designed. 

Many attempts have been made to design a computer 
architecture that is more suited for picture processing 1-7 and 
in some cases the machines have also been constructed. 

The design of the PICAP processor has evolved through 
a number of generations. Starting with a simulation study in 
1971, 8 the first version of the processor was built in 1973. 4 
Gradually as experience with the processor accumulated 
new features were incorporated such as automatic histo- 
gramming and collection of various measurements made on 
the picture. The final version of PICAP which will be de¬ 
scribed here has been operational since the beginning of 
1975. 

PICAP AND ITS ENVIRONMENT 

The following paragraphs give a brief presentation of the 
PICAP Picture Processing Laboratory as a background to 
the description of PICAP’s architecture. For a complete 
description of the laboratory see Reference 9. 

PICAP operates in close connection with a conventional 
minicomputer of quite standard configuration. Standard pe- 
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ripherals, such as alphanumeric and graphical displays, and 
a multi-user disc operating system are the main features. 
PICAP and the computer are also connected to an advanced 
television input/output interface, TIP 15 (see Figure 1). Both 
PICAP and TIP are controlled by the computer which be¬ 
sides its controlling function performs all computations of 
non-pictorial nature. 

Through TIP PICAP has access to an entire TV field of 
512x640 pixels from which a randomly positioned window 
of 64 x 64 4-bit pixels may be extracted at full video rate. 
Both the sampling density and gray-level range in the digi¬ 
tization process are controlled by a set of programmable 
parameters. Thus the window may in one frame represent 
the entire field coarsely sampled with the full dynamic video 
range mapped onto the 16 level gray-scale and in another 
frame it may represent a small subpicture densly sampled in 
a certain slice of the video range. Consequently a scene may 
be analyzed on many levels starting with a coarse overview 
and ending with a detailed analysis in full resolution. These 
features correspond in a way to the vision systems found in 
nature where only a limited retinal area, the fovea, have 
high spatial resolution and where a scene is sequentially 
scanned by mechanical eye movements. 

There are four video channels available in TIP of which 
two are used for the present. One camera is permanently 
mounted on a light microscope which is supplied with a 
computer controlled stage scanner and the other is mounted 
on an optical bench so that ordinary slides may be read. 

THE ARCHITECTURE OF PICAP 

PICAP belongs to the category of “single-instruction- 
stream multiple-data-stream” (SIMD) processors. Unlike 
many processors used for pattern recognition, it is fully 
programmable with an instruction set especially tailored for 
efficient processing of both gray-level and binary pictures. 
Equally important as the feature of programmable picture 
into picture transformations is PICAP’s capability to extract 
measurements. In all pattern recognition tasks the (often 
very redundant) input data has to be reduced into a set of 
adequate descriptors that describe its relevant features, such 
as gray-scale distribution, object position and shape. It is 
very important that this can be done efficiently since these 
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Standard 

minicomputer 

configuration 


Figure 1—Processor environment 

descriptors, measurements, form the basis for a subsequent 
classification. From the processor’s point of view the meas¬ 
urements constitute the final result. The picture transfor¬ 
mations only serve to enhance the features that finally are 
to be measured. 

The 64 x 64 4-bit window mentioned in the preceding sec¬ 
tion constitutes the basic data unit that PICAP operates on 
much in the same way as the 16-bit word is the data unit for 
common minicomputers. Figure 2 shows schematically the 
two main types of processor action on the picture data, 
multiple picture processing and neighborhood processing. 
In the multiple processing mode two, X and T, or more (up 
to nine) pictures are operands giving a result Z, the pixels 
of which are determined by the corresponding pixels in X 
and T and the actual operation. Thus 

Zy = F c (xy, Yfj, . . .) i,j= 1, 2, . . . , 64 

The mapping function F c is defined by C, a set of parameters 
which is supplied by the instruction. Typical multiple mode 
instructions are weighted summation (e.g., subtraction) of 
pictures and masking. 




M 

Figure 2—Schematic operation of the processor 



The neighborhood operation mode applies to a single pic¬ 
ture, X, at a time. In this case the output pixels of array Z 
are determined by corresponding pixels in X and their im¬ 
mediate surrounds, the 3x3 neighborhoods N(x tj ). 

Zy=F c (N(Xjj )) i,j= 1, 2, . . ., 64 

The actual neighborhood operation is determined by the 
parameter set C and it is performed identically on all neigh¬ 
borhoods of X. Convolutions, shrinking and labelling of 
pictures are typical examples. 

Note that the actual mapping, symbolized by F c , is iden¬ 
tical in the two modes. It is only the operands that differ. 

In addition to the instructions belonging to the categories 
above there are trivial instructions, such as input/output and 
shift of pictures, that will not be discussed here. 

The main units of the processor are shown in Figure 3. 
The picture memory PM is a set of nine separate picture 
registers R 0 , R t , . . . , R s into which any window of the 
input TV field may be loaded. The registers are also used 
for intermediate results much in the same way as the reg¬ 
isters of common CPUs are. The MLA unit in which the 
actual processing is performed takes its operands either from 
the nine picture registers (multiple mode) or from the neigh¬ 
borhood registers Q 0 , Q lt . . . , Q 8 of the LB unit (neigh¬ 
borhood mode). The parameter that determines the actual 
processing resides in CTM which essentially is an extension 
of the instruction register not shown in the figure. 

Simultaneously to the processing of pictures the MLA 
unit collects instruction dependent measurements that are 
stored in a set of thirty-three registers, MR. To enable 
conditional branches in the instruction flow to the processor 
the contents of these registers are made available to the 
controlling computer at the end of each executed instruction. 
In a way the measurement registers are the analogue of the 
status register in conventional computers in that they de¬ 
scribe the outcome of an instruction. 

As has been stated above PICAP is a SIMD processor 
since a single instruction determines the operation per¬ 
formed on a large set of operands, the pixels. Physically, 
SIMD processors may be designed with different degrees of 
hardware parallelism and this is especially apparent for pic¬ 
ture processors where there are three distinct levels on 
which the parallelism may be introduced, the pixel, the 
neighborhood and the picture levels. On the pixel level there 
is a choice on the number of bits that can be processed in 
parallel. In PICAP this is four, contrary to most existing 
picture processors that are binary. On the neighborhood 
level the degree of parallelism is equal to the number of 
neighborhood pixels that may be accessed and processed 
simultaneously which in most existing processors including 
PICAP is nine, a 3x3 neighborhood. Finally on the picture 
level the choice is on the number of neighborhood process¬ 
ing units. In PICAP there is one, the MLA unit, which 
means that the neighborhoods are processed sequentially. 
The level on which the great hardware effort should be made 
is a controversial question which will be commented on in 
the last section. 
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Figure 3—Matching logic, adder and line buffers 


THE INSTRUCTION SET 

There are two main PICAP instruction categories: the 
logical and the arithmetical instructions. The instructions 
are described in detail in References 9 and 10 and only a 
summary will be given here. Their characteristics will be 
described as applied in the neighborhood operation mode. 
The implications for the multiple mode operation are ob¬ 
vious. 

The logical neighborhood instructions are based on gen¬ 
eralized 3x3 templates called condition templates consisting 
of nine subconditions one for each of the neighborhood 
elements. Each instruction may contain up to eight such 
condition templates. In the course of execution the picture 
neighborhoods are all subjected to a test whether they do or 
do not belong to the class of neighborhoods defined by the 
set of condition templates of the instruction. If a neighbor¬ 
hood matches a certain template its center pixel is given a 
new value, T, that is supplied by the instruction. All other 
pixels remain unaffected. In case of multiple hits a simple 
precedence rule determines the new value. The entire trans¬ 
formation may be thought of as an associative multiple 
search in two dimensions for pixels whose neighborhoods 
correspond to certain features as defined by the templates. 


The subconditions C K are composed of a relation R K and 
a number n K , C K =(R K , n K ) .The condition is fulfilled iff the 
pixel q K is related to n K by R K . Four different relations may 
be used: equal to ( E ), less than ( L ), greater than (G) and 
the universal relation (A= don’t-care). The subconditions Eire 
defined over the 3x3 neighborhood forming a condition tem¬ 
plate 


C4 C3 C2 
C—C 5 Co Ci T 
C 6 C7 Cg 


A certain neighborhood matches a condition template iff 
all its element fulfill their corresponding subconditions. To 
each condition template there is associated a transition value 
T, the value into which the center pixel is transformed in 
case of a match. 

The format of the instruction is shown in Figure 4. It is 
a variable length format as the number of condition template 
is not fixed. The instruction-head contains a modifier field 
and the addresses of the source and destination registers 
besides the operation code. A number of variations to the 
instruction’s basic properties as described above may be 
specified in the modifier field (see Figure 5). For example, 
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Figure 4—Format of logical instructions 


the modifier M determines the mode of operation, multiple 
or neighborhood mode. The other modifiers apply to iso¬ 
tropic application of the templates (R4 and R8), loop oper¬ 
ations (It), extraction of the first matching neighborhood 
(In) and sequential picture operations (S). A full descrip¬ 
tions of these are given in Reference 10 which also contains 
a number of typical operations. A trivial example is given 
below. 

Example 1: Neighborhood dependent thresholding. 


X 

Gn 

X 


X 

X 

X 

Gn 

Gn 

Gn 

1 

X 

X 

X 

X 

Gn 

X 


X 

X 

X 


The first template of the instruction covers the class of 
five element neighborhoods whose elements have a value 
greater than n (Gn). Due to the precedence rule, center 
points that belong to such neighborhoods will aquire the 
value 1 irrespective of the second template. As the latter 
covers all possible neighborhoods the remaining pixels will 
get the value 0. (End of example.) 

Compared to the logical instructions the arithmetical ones 
are conceptually much simpler. They have the form of a 
convolution between an array of coefficients defined over 
a 3x3 neighborhood and the picture array. The computa¬ 
tions are followed by normalization with a factor 2~ n so that 
the result will not overflow. Hence the center pixel of the 
neighborhood is replaced by the value of the expression 

QS 2~ n 


The coefficients {ntt} and the normalizing exponent n have 
eight and four bits precision respectively. The instruction 
format is similar to the logical instruction format. 

The following example shows a simple contrast enhance¬ 
ment operator. 

Example 2: Contrast enhancement. 

Coefficient array: Normalizing factor: 


-1 

-1 

-1 

-1 

40 

-1 

-1 

-1 

-1 


2~ 5 (1/32) 


The operator has unit again for low spatial frequencies and 
enhances high frequencies. 


MEASUREMENT EXTRACTION 

The picture transformations described in the preceding 
section are accompanied by a fairly large set of automatically 
performed measurements. 

• Neighborhood counting 

• X-Y extent determination 

• Gray-scale maximum, minimum and average 

• Histogram collection 


Neighborhood counting 

To each condition template of a logical instruction is as¬ 
sociated a counter which is incremented for each occurrence 
of a matching neighborhood. Hence locally countable prop¬ 
erties such as Euler number, area and perimeter are easily 
extracted. 


X-Y extent determination 

To provide for extraction of positional information there 
is a set of four registers X min , X max , T min and T max that gives 
the maximum and minimum coordinates on both axes for 
the occurrence of a matching neighborhood. The set of co¬ 
ordinates defines the minimum rectangle aligned with the 
axes that enclose a set of features which is described by the 
templates. 


2_7 


M 

R4 

73 

CD 

It 

In 

S 



modifier field 

sequential operation 
interrupted operation 
iterative operation 
isotropic operation (8) 
isotropic operation (4) 
multiple operands 


Figure 5—The modifier field 


Grayscale maximum, minimum and average and 
histogram collection 

The gray-level distribution and its maximum, minimum 
and average are collected in a set of 19 registers HC 0 , HCl, 
. . . , HC lb , G max , G min and G av . Any transformation of a 
picture results in an updated set of gray-scale measurement. 

It should be noted that the inclusion of automatic mea¬ 
surements is especially simple in a processor where the pic- 
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ture is sequentially processed since it amounts to little more 
than a straightforward counting procedure. 


EVALUATION AND PERFORMANCE 

During the years that PICAP has been operational it has 
been employed in many different pattern recognition proj¬ 
ects. A subset of these form the basis for the following 
evaluation of the instruction set. The subset is chosen to 
contain quite dissimilar applications in order to reflect the 
operational use of PICAP in different contexts. The evalu¬ 
ation is based on three programs: printed circuit board in¬ 
spection, malaria parasite detection and fingerprint coding. 
The object of the statistical analysis below has been to 
quantify the use of the instruction categories, modifiers etc. 
and most of all to see how often the controlling minicom¬ 
puter has been used for picture processing. 

The printed circuit inspection program tests the circuits 
with respect to the minimum tolerable conductor widths and 
separation. It incorporates necessary adaptive thresholding 
of the gray-level input pictures 11 and generation of a pseudo 
color output picture that indicates the defects. The nominal 
width of the conductors and insulators is checked by an 
appropriate number of erosion and swelling operations. The 
statistics are shown in Figure 6. 

The statistic material shown in Figure 7 is the pooled 
statistics from two separate programs for classification of 
and core location in fingerprints. For a complete description 
of these programs see References 12 and 13. The fingerprints 
are read by a TV camera. Through the use of appropriate 
direction sensitive filter operators the ridge and valley pattern 
is extracted. Then syntactical methods are used to compute 
the category assignment and the location of the fingerprint 
core. 

The malaria parasite detection program 14 is the most ad¬ 
vanced of the application programs. Biological specimens of 
malaria-contaminated blood cells in a thin smear on a mi¬ 
croscope slide are analyzed. During the scanning procedure 
the infected blood cells are located and the parasites are 
classified according to their stage of evolution. One test 
evolves the analysis of 5000-6000 red blood cells which takes 
approximately five minutes for a skilled microscopist 
whereas the automatic program takes eight minutes. Figure 
8 shows the statistics. The descriptions above are of course 
very schematic and do not at all give full credit to the actual 
sophistication of the programs. 

It is apparent from the pooled statistics in Figure 9 that 
the predominant instruction category is the logical instruc¬ 
tion. This category accounts for over 70 percent of all exe¬ 
cuted instructions. It is less dominant in the case of the 
malaria parasite program (55 percent) presumably because 
the source data is inherently non-binary. The picture transfer 
and arithmetical instruction types are equally frequent, ac¬ 
counting for approximately 15 percent each, whereas the 
shift instruction is virtually never used. As a shole the ma¬ 
laria parasite program exhibits greater balance between in¬ 


struction types, reflecting the general type of processing 
required. 

The modifiers of the logical instructions are used in ap¬ 
proximately 50 percent of the cases which means that the 
nonmodified, i.e., the one-operand local, logical instruction 
accounts for 50 percent. Among the modifiers the isotropic 
(RS), and the multiple operand modifiers are used most 
frequently. Of the eight available templates in the instruction 
format only one or two are needed in nearly 90 percent of 
the cases. The one- and two-template instructions account 
for 44 percent each. 

The picture transfer instructions, of which the input type 
is predominant, are second-most-frequently used. It is an 
interesting fact that of all executed instructions input from 
the TV field to PICAP accounts for over 10 percent which 
means that on the average every tenth instruction inputs a 
new picture. This emphasizes the need for fast transducers. 
Loading of a picture should not be slower than the execution 
of one instruction if the processor is to be kept busy 90 
percent of the time which seems to be a reasonable figure. 
Another interesting fact is that transfers of pictures to the 
host computer’s memory are virtually nonexistent (less than 
1 percent) which means that essentially all picture process¬ 
ing operations are performed in PICAP. As PICAP pro¬ 
grammers always have the option of processing the pictures 
in the host computer this indicates that the chosen instruc¬ 
tion and measurement sets are very well adapted to the task 
set. 

The arithmetical instructions are used to approximately 
the same extent as the transfer instructions, 14 percent. 
Multiple operand instructions such as addition/subtraction 
of pictures are relatively infrequent, 12 percent. 

Among the picture registers R$ is most frequently used. 
In 68 percent of the instructions Ro appears as either source 
or destination register. R 0 and R x together account for over 
80 percent of the cases which indicates that the number of 
nine registers is more than sufficient. 

To measure the performance in terms of the execution 
time for the basic instructions is always unsatisfactory since 
it obviously depends on many other factors such as the 
speed and versatility of the picture transducer, the host 
computer and its operating system. The best figure of merit 
for the PICAP system is given by the malaria parasite pro¬ 
gram which, as has been stated above, has a performance 
of a skilled microscopist. The processing speed including 
input, feature extraction and classification of the parasites 
is 10-15 blood cells per second. 

Finally, for the sake of completeness the execution time 
for the basic one-template and two-template logical instruc¬ 
tions are 1.2 and 1.5 jtts/pixel respectively. The execution 
time for arithmetical instructions are 1.2 and 3.6 /xs/pixel for 
a smoothing and an arbitrary convolution respectively. 


CONCLUSIONS 

The PICAP architecture and instruction set has been shown 
to cover the needs for solving typical pictorial pattern rec- 
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ognition applications very well. The performance in an ad¬ 
vanced task like malaria parasite detection, equals that of a 
skilled microscopist. 

The fact that on the average only nine operations are 
performed on each input picture is not explained by the 
well-adapted instruction set alone. It is also a consequence 
of the frequent use of several levels of sampling density. 
This means that featureless parts of the input scene often 
may be located in a coarse survey at an early stage of the 
recognition procedure. Since such parts need not be further 
processed the average number of operations per picture is 
lowered. 

The high frequency of input instructions stresses the need 
for fast picture transducers. On the other hand, the speed 
requirements of a processor with PICAP’s characteristics is 
limited to not much more than nine times the speed of the 
system’s transducer. 

The ability for fast collection of various measurements is 
very important. The ease with which such facilities are 
incorporated in PI CAP, using not much more than a simple 
counter for each measurement, speaks strongly in favor of 
physically sequential machines. 

In processors where parallelism on the picture level is 
used, corresponding measurements have to be extracted by 
a cumbersome programmed procedure. The number of pro¬ 
gram steps for such an extraction may be several orders of 
magnitude larger than the initial number of processing steps 
that produced the feature to be extracted. Therefore the high 
inherent speed of truely parallel machines in picture-picture 
transformations is very much impaired in pattern recognition 
applications. 

In fact, PICAP with its low order of complexity can in 
several cases compete with more elaborate parallel designs. 
Another reason for this is that the sole neighborhood proc¬ 
essing unit of a sequential processor may easily be made 
more powerful than the corresponding multiple units in a 
parallel design. 

Finally, the continuing advance of IC manufacturing 
makes the future look bright for practical, high-performance 
picture processors. 
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Design of local parallel pattern processor for image 
processing 
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INTRODUCTION 

Several image processing machines have been designed and 
some of them already implemented. The Illiac III by Mc¬ 
Cormick, 1 hexagonal processor GLOPR by Golay, 2 and bi¬ 
nary image processor BIP by Gray 3 are examples. The idea 
of parallel processing hardware has proved its feasibility, 
appealing to a number of industrial applications. Special 
purpose image processors, such as the optical character 
reader, 4 the white blood cell differential count analyzer, 5 the 
remotely sensed data analyzer 6,7,19 and the computer tom¬ 
ography machine 8 have their characteristic architectures ac¬ 
cording to their purpose. General purpose image processors 
have been proposed by several research groups 9-11,18 and 
are demonstrating their ability. 

The problem now to be solved is determining what archi¬ 
tecture of image processing machines is superior in the cost 
performance ratio. Trade offs in various aspects like speed, 
generality and economy, should be discussed. Our approach 
is a microprogrammable local parallel pattern processor 
(PPP) which has several basic image processing functions in 
a simple hardware. In the second section, major concepts 
considered for the design of the PPP will be discussed, and 
in the third section, the details of the implementation of the 
basic image processing functions are described. Some simple 
examples of programming the PPP will be shown in the 
fourth section, but the details of command formats, appli¬ 
cation softwares for remotely sensed data analysis, medical 
image processing, etc. can be found in other papers. 12-14 

DESIGN CONCEPTS 

The PPP has several features in design concepts for im¬ 
proving the image processing cost performance ratio. These 
features contribute to speeding up major calculations in 
image processing to approximately 100 times faster than by 
the conventional digital computers, and to reducing the 
hardware cost and complexity. 

Image memory 

The PPP is directly connected to the image memory unit 
of the interactive image processing system TOSPICS (To¬ 


shiba Pattern Information Cognitive System) shown in Fig¬ 
ure 1. In the TOSPICS, the image memory unit plays five 
roles, as high speed buffer memory for image input/output 
devices, refresh memory for color display monitors, working 
memory for the PPP, buffer memory for a magnetic disk 
unit and external bulk memory for the host mini-computer. 
This configuration minimizes the “idle” data transfers 
among hierarchical levels of the memory. The time for those 
data transfers generally share a large fraction of the total 
throughput time by conventional computer configurations, 
which are not necessary for the essential calculations in the 
image processing. 

Basic image processing functions 

There are several types of computation that are frequently 
used in image processing. Most of such operations take a 
large amount of time. The parallel image processing machine 
should be fast enough to execute those basic operations for 
improvement in the cost performance ratio. 

Those basic image processing functions are listed as fol¬ 
lows: 15 

(1) Two Dimensional Convolution 

m n 

G(X,Y)=2 s WyFijix, y) (1) 

i—1 J=1 

where F i} (X, Y ) is a neighboring pixel of input image data 
around the coordinate value (X, Y), and W i5 is an element 
of weighting matrix of size mxn. This computation is used 
in such image processing operations as: convolution, spatial 
filtering, general linear algebra computations, correlation, 
template matching, texture analysis, K-L transformation of 
image, 13 and scene classification by matched filter or maxi¬ 
mum likelihood method. Many operations used in spatial 
frequency domain analysis, such as restoration, enhance¬ 
ment, reconstruction from projections can be similarly ex¬ 
pressed by Eq. (I). 16 

(2) Point Mapping 

G(X,Y)=f{F(X,Y)} (2) 

where 0 is a single-valued (usually nonlinear) function. Point 
mappings are used for such operations as: enhancement, 
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(o) TOSPiCS Overall View 



(b) TOSPICS Blockdiagram 
Figure 1—Overview and blockdiagram of TOSPICS 


pixel processing elements, which works simultaneously on 
independent data stream per processing element. However, 
the image processors on the fully-parallel basis using a large 
number of LSI calculation elements 17 have restricted data 
communication between only adjacent neighboring pixels. 
This architecture will limit the applicability of the image 
processing operation of Eqs. (1) and (3), or will need exces¬ 
sive time to access neighboring image data, which cancels 
the speed-up effects brought about by the fully-parallel 
basis. 

A pixel generally has stronger dependency to neighboring 
pixels than to those in the distance. The range of neighboring 
pixels depends on the nature of operations, but it falls be¬ 
tween 3x3 and 32x32 pixels surrounding the center pixel. 
In the PPP, the local parallel processing, i.e., parallel ac¬ 
cessing and calculation within local dependency up to nXn 
neighboring image data, is carried out in the specially de¬ 
signed circuits. Then, the center position of local parallel 
operation is shifted sequentially to the next position. The 
combination of local parallel processing and the sequential 
processing fits the serial data transfer characteristics of 
image data input/output devices and image memory. 

Dynamic microprogramming 

The PPP includes seven special purpose modules for two 
dimensional convolution, point mapping, linear coordinate 
transformation, logical filtering, histogram generation, re¬ 
gion labeling and pixel operations. These functions can be 
combined for more global and complex functions such as 
texture analysis, shape identification, region separation and 
so on. By using dynamically rewritable microprogram ar¬ 
chitecture, powerful and flexible programs of image proc¬ 
essing were designed and demonstrated in the interactive 
image processing system TOSPICS. 12 * 13 

IMPLEMENTATION 


nonlinear filtering, gray scale mappings, multilevel thresh¬ 
olding, pseudo-color generation, correction of intensity non¬ 
linearity of image sensors and displays. 


(3) Linear Coordinate Transformation 


G(X,Y)=AF(X,Y) 


(3) 

where A is affine transformation specified by a linear co¬ 
ordinate transformation matrix and shift values. This oper¬ 
ation is used for rotation, size change and registration of 
image. 

In addition, logical filtering, region labeling, histogram 
generation and pixel operations are frequehtly used. These 
basic image processing functions are greatly speeded up by 
the use of specially designed circuits. 


Local parallel vs. fully-parallel 

There are two types of parallelism in image processing. 
The most popular idea is a two dimensional array of identical 


System configuration 

An overall blockdiagram of the PPP is depicted in Figure 
2. The host computer in TOSPICS is a minicomputer TOS- 
BAC-40C with 64 K bytes memory. The image memory, the 
TOSPICS pivot, consists of four layers of 512x512 pixels, 
8 bit per pixel, and four planes of 512x512 binary graphic 
memory. The two dimensional convolution module includes 
a buffer memory of a shift register and a weighting matrix 
memory. The shift register storing the image data of n lines 
enables a parallel accessing of «x n image data for the local 
parallel calculations. The 512 byte scratch-pad memory is 
needed for table look-up operations in logical filtering, point 
mapping and region labeling. It is also used as 256 registers 
of 16 bit length in histogram generation. 

The image memory is connected to the high speed bus in 
the PPP, which has a transfer rate of 4 M bytes per second. 
The interface to the image memory has two address con¬ 
trollers which independently calculate the input and output 
memory addresses and transfer image data simultaneously 
to or from any image memory layer or graphic plane. Those 
address controllers are used for the linear coordinate trans¬ 
formation of image data. They are capable of calculating 









1027 


Design of Local Parallel Pattern Processor for Image Processing 



Figure 2—Local parallel pattern processor blockdiagram 


next memory addresses automatically, and checking the 
boundary of image data. 

The microprograms and execution commands are trans¬ 
ferred through the interface from the host computer with a 
set of parameters to execute those programs and commands. 
The processed image data are usually transferred to an image 
output device, but data can be also stored in the disk mem¬ 
ory for the higher level processing, such as image under¬ 
standing. 

Table I shows a repertoire of execution commands. They 
are grouped in three categories: initialization commands, 
parameter transfer commands and function execution com¬ 
mands. Initialization commands initiate the internal status 
of the PPP and transfer any user’s microprograms into a 
random access program memory. Some of the frequently 
used microprograms, such as enhancement, pixel operations 
and maximum likelihood decision can be stored in the read¬ 


only program memory. Parameter transfer commands permit 
parameter data communication between the host computer 
and the PPP. Function execution commands start image 
processing functions. At the end of operations, an interrupt 
signal is generated to the host computer. 


Details of the basic function modulus 

Two dimensional convolution module 


Figure 3 shows a mechanism for two dimensional con¬ 
volution function by Eq. (1). It includes m 2 multiplications 
and (m 2 -l) additions for the weighting matrix W u whose 
size is rnX/n. It is obvious that the more parallel arithmetic 
elements are used, the faster the computation will be per¬ 
formed and the more circuit cost and complexity increase 
exponentially. To reduce the number of required arithmetic 
elements, a time-shared pipeline control technique is 
adopted. The principle of the control is as follows: Eq. (1) 
is expanded as 

m 

G(X,Y )=2 W u F(X+n-\,Y+n-j) 

j=i 


+ - + I W mi F(X+n-m,Y+n-j) (4) 

i=i . 


where m is the size of the weight matrix w ih F(X, Y) is a 
pixel of input image data at the coordinate values (X, Y), 

and n= 

4, each 

lated by m multipliers and (m-1) adders in parallel, while 
m terms in Eq. (4) are calculated successively and summed 
up by the last adder. This configuration of two dimensional 
convolution module reduces the number of multipliers and 
adders from m 2 to m, while the total calculation time in¬ 
creases m times, compared with the folly-parallel base con¬ 
figuration. 

In practice, the weighting matrix size was determined to 
be eight as a result of a trade-off between speed and econ- 


'm+li 


. By the use of the circuit shown in Figure 
term of Eq. (4), a partial sum of products, is calcu- 


TABLE I.—Command Repertoire 


Mnemonic Code 

Function 

CLR 

PPP Initialization 

LMP 

Load microprogram in the program memory 

LOT 

Load data in the scratch-pad memory 
or in the weight table 

SAR 

Set parameters in address registers 

ASM 

Assign memory for processing 

TDT 

Transfer data from the table memory 

FLT 

Two-dimensional convolution 

AFN 

Affine coordinate transformation 

HST 

Histogram computation 

DC V 

Data conversion 

LFL 

Logical filtering 

LBL 

Region labeling 

PIX 

Pixel operation 


FILTER WEIGHT MATRIX 



INPUT IMAGE 

Figure 3—Two-dimensional convolution mechanism 
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Figure 4—Two-dimensional convolution circuit blockdiagram 


omy. Every partial sum of products is calculated at the clock 
rate of 125 nanoseconds, therefore one micro second is 
needed to complete the calculation of Eq. (1) for output 
pixel G(X, y). This local parallel calculation accomplishes 
64 multiplications, 64 additions, and 256 data accesses within 
1 micro second. Thus, the input and output data transfer 
rate between image memory and the PPP is 1 M bytes per 
second. 

In addition, for simplifying data access, a long shift reg¬ 
ister, which buffers seven lines of image data is used in the 
two dimensional convolution module, as in Figure 4. A par¬ 
tial column of input image data F(X+n-l,Y+n-j), 
j= 1,2,. . . ,ra is used for the calculations of G(X, Y), 
G(X+\,Y), . . . , G(X+m,Y). If the partial sums of prod¬ 
ucts for those partial column image data in G(X, Y), . . . , 
G(X+m, Y) are calculated when the column data are shifted 
at the end of the buffer shift register simultaneously, then 
the data access time to fetch image data to the arithmetic 
units can be made nil. Partial sums are sequentially accu¬ 
mulated into the accumulator registers to complete the cal¬ 
culation for output image data. 

Logical filtering module 

Logical filtering is an operator applied to binary image 
data for thinning, shrinking, swelling, boundary detection, 
topological feature extraction operation etc. Figure 5 depicts 
a result of the logical filtering operation that extracts simul- 

INPUT BINARY PATTERN FILTERED OUTPUT 



Figure 5—Logical filtering operation result 


taneously from a thinned binary image, end points ( E ), 
branchingpoints(Xory),linesegmentdirections(l,2,. . . ,8), 
isolated points (/), and small loop points (0). A logical fil¬ 
tering module takes up every 3x3 submatrix in the image 
data, and then converts the 9 bit binary data to correspond¬ 
ing output code, consulting the conversion table stored in 
the scratch-pad memory. The 9-bit binary data are used as 
a binary address for table look-up of the scratch-pad memory 
with a 512 byte capacity. 

Linear coordinate transformation module 

Two dimensional linear coordinate transformation (affine 
transformation) is performed by the address control module. 
Eq. (3) in the earlier section can be rewritten as 

X=pX' + qY'+Xo 
Y=rX' + sY'+Yo 

where X and Y are the input image coordinates and X' and 
Y' are the output image coordinates. In digital image proc¬ 
essing, image data are expressed in the form of positionally 
sampled data, so X, Y, X’ and Y' are to be integer values. 
All values in the address control module are expressed in 20 
bits with 9-bit fraction. In practice, as the values of X' and 
Y' change in raster mode, input image coordinates X and Y 
are calculated by Eq. (5). Since the computed values of X 
and Y are not always integer, the address control module 
should interpolate the approximated gray level at coordi¬ 
nates (X, Y) from several surrounding image data, which 
have integer coordinate values near (X, Y). 

Three interpolation algorithms are built in the PPP: near¬ 
est neighbor method, bilinear interpolation method and 
cubic spline function method. The address control module 
has additional registers to calculate the addresses of neigh¬ 
boring image data points at high speed. Other interpolation 
algorithms, such as raised cosine method and two dimen¬ 
sional sine function method, can easily be micropro¬ 
grammed. Moreover, nonlinear coordinate transformation 
can be attained by the algorithm using linear approximations 
for segmented image regions. 

The speed of the affine transformation for 512x512 pixels 
was 0.26 second by the nearest neighbor method, 1.04 sec¬ 
ond by the bilinear interpolation, and 4.16 second by the 
cubic spline function method, which uses sixteen neighbor¬ 
ing image data points. 

Scratch-pad memory and histogram counter 

Point mapping of Eq. (2) is done with the scratch-pad 
memory in Figure 2, where function </> is stored in tabular 
form to be looked up. Because the gray levels of input image 
data are expressed by 8 bits, the output value corresponding 
to any input gray level can be stored in the 256 byte capacity 
scratch-pad memory. 

The scratch-pad memory is also used for histogram count¬ 
ing registers for image data. In this case, it sums up registers 
to count the frequency of points with the same gray level in 
the input image data specified area. From limitation of the 
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INPUT BINARY PATTERN LABELED OUTPUT 


Figure 6—Region labeling operation result 

scratch-pad memory capacity (256x2 bytes), HST (Histo¬ 
gram) command is applied to 256x256 image data at a time 
for a monotonous image. However, it can be applied to 
512x512 image data when they are full of variety in gray 
levels. 

Region labeling module 

The region labeling module partitions a binary image data 
into several disjointed blocks by testing the connectivity in 
a 3 x2 submatrix as shown in Figure 6. It is possible to select 
one of two definitions of connectivity: (a) four direction 
connectivity, such that if any two “1” points are horizontally 
or vertically adjacent in image data, after which they are 
decided to be connected, and (b) eight direction connectiv¬ 
ity. In the example of Figure 6, a four direction connectivity 
mode is applied. If the eight direction connectivity mode is 
used, regions A and B would be decided to be connected 
and assigned by the same label. 

Pixel operation module 

Beside the most frequently used image processing func¬ 
tions mentioned above, the other problem-oriented opera¬ 
tions, such as nonlinear coordinate transformation, maxi¬ 
mum likelihood classification and texture analysis are 
dynamically microprogrammable. Microprograms are trans¬ 
ferred from the host computer to the microprogram memory 
in Figure 2 by LMP command and executed by PIX com¬ 
mand. Any arithmetic and logical operations for each pixel 
are interpreted by the microprogrammable controller (/jlC). 
The ixC functional roles are as follows: 

(1) Pixelwise operation for image data 

(2) Communication to the host computer 

(3) Interface control to the image memory 

(4) Internal data bus control 

(5) Control of each functional module. 

The word length of a microprogram instruction is 40 bits. 
The ixC cycle time is 250 nanoseconds. 


TABLE II.—Applicable image processing repertoire 


Basic Function 

Applications 

Two-Dimensional Convolution 

Smoothing, Noise averaging, Enhancement, 

Differentiation, Restoration. 

Affine Coordinate Transform 

Enlargement. Shrinking, Rotation. Shift, 

Geometric Correction . Mosaicking, 

Logical Filter 

Thinning , Noise Clearing. Gap Filling, 

Feature Detection, 

Region Labeling 

Region Separation. Region Coloring, Area Counting, 

Data Conversion 

Y- correction, log/exp, sin/cos, x 2 //x 

Thresholding, Histogram Equalization, 

Histogram Generation 

Gray Level Frequency Counting , Area Counting, 

Pixel Operation 

Pixewise arithmetic and logical operations, 

Ratioing. Shading Correction, 


APPLICATIONS 

The PPP has been proved to be a powerful and economical 
tool for research and experiments in image processing, es¬ 
pecially when it is used in an interactive image processing 
system. It should be emphasized that the PPP not only has 
the ability of gray level image processing but also of binary 
image processing. 

A number of subprograms and application programs, using 
the PPP in TOSPICS, have already been developed for med¬ 
ical image processing, remote sensing and industrial image 
processing. Table II shows a list of repertoire of applicable 
image processings by PPP’s basic functions. Each basic 
image processing function can be carried out at the rate of 
1 microsecond per pixel. Some examples of execution time 
required for image data of 512x512 pixels are shown in 
Table III. 

A typical program sequence, using the PPP, would have 
the following steps in the host computer: 

Step 1. Assign image memory for input and output 

Step 2. Set parameters in the address controllers 


TABLE III.—Some examples of execution time for 512x512 Pixels 


Function 

Applications 

Execution Time 

Two-Dimensional 

Convolution 

Laplacian. Smoothing, etc 

262mS 

Logical Filter 

Feature Detection 

Thinning 

262mS 

262xW°mS 

Region Labeling 

Region Separation 2) 

Particle Measurement 31 

524mS 

786mS 

Data Conversion 

V - Correction 

Thresholding 

262mS 

Histogram Generation 

Frequency Counting 

262mS 

Affine Coordinate 
Transform 

Enlargement with Rotation 
Axis Skewing 

2 62mS 

Pixel Operation 

Logical Operation 
Arithmetic Operation 

524mS 

Gradient Operation 

Edge Detection 

2.36S 


1) W is maximum lint width 

2) No of separated regions is less than 254 

3) Region separation and area counting 
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Step 3. Specify the basic function or transfer the micro¬ 
program string 

Step 4. Command execution start 

Step 5. Sense the end of execution from the PPP. 


The gradient operation to image data, which calculates 
the magnitude and direction of gradient vectors, is one of 
the suitable application examples to show an assortment of 
the basic functions. 

If the input image is denoted by F(Z, F), then the mag¬ 
nitude G(X,Y) and direction code D(X,Y) of the gradient 
vector at point {X, Y) can be obtained as, 




( 6 ) 


D(X, F)=0 tan" 1 


(dF(X, Y) 

/3F(Z,F)n 

d Y / 

dX j 


where dF(Z, F)/dZ and dF(X, Y)/d Y are output values of 
the two dimensional convolution with Z-derivative filtering 
and F-derivative filtering, respectively. ©(0) indicates a 
multilevel thresholding function which encodes an angle 6 
into one of 16 direction codes. 

Let the input image be stored in Ml memory layer, the 
main stream of subprogram to calculate G(X, F) and D(X, F) 
is as follows: 


More global image processing functions, such as texture 
analysis, shape detection, region separation, etc., can be 
microprogrammed in combination with these basic func¬ 
tions. 

Some experimental results have been demonstrated for 
remotely sensed data and medical image data. These dem¬ 
onstrations indicate that the PPP is sufficiently powerful and 
economical enough to explore novel applications and quick 
analysis in the image understanding studies. 

A further expansion of the PPP, which is currently being 
developed, is to realize the fast Fourier transformation and 
some vector operations in a simple hardware. 
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(a) Execute spatial filtering with Z-derivative weighting 
matrix, and store the results in M2 

(b) Execute F-derivative operation for Ml image data and 
store the results in M3 

(c) Using the point mapping function of x 2 , calculate the 
square of M2 and M3, and add these values to M4 

(d) Using pixelwise operation of division, calculate M2/ 
M3 and store the results in M3. Then, execute two 
point mapping operations, tan -1 * and 0(0), succes¬ 
sively for M3. The direction code D(Z, F) is obtained 
in M3. 

(e) Using point mapping operation of Vx for M4, calcu¬ 
late the magnitude of gradient vector G(X,Y) and 
store results in M2. 

The concepts and algorithms of an image processing pro¬ 
gram for the parallel image processing machine may differ 
from those of conventional programs implemented in serial 
digital computers. It is an open problem to develop image 
processing, such as the relaxation method by Rosenfeld 20 
and the primal sketch by Marr. 21 

CONCLUSIONS 

A microprogrammable local parallel pattern processor (PPP) 
has been proposed and realized. The PPP has capabilities of 
programming local parallel basic operations, coordinate 
transformations and pixel operations. The processing time 
is a quarter second for 512x512 pixel image, which is about 
two orders of magnitude faster than conventional serial com¬ 
puters. 
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INTRODUCTION 

Recent progress in architecture technology has had a great 
impact on the study of pattern recognition problems. The 
increase in processing power is bringing a new concept of 
“understanding” into recognition areas. Recognition objec¬ 
tives have been greatly extended to meet real social require¬ 
ments. As observed in the speech recognition system de¬ 
veloped at Camegie-Mellon University, for example, efforts 
have been shifted from attaining higher recognition of iso¬ 
lated words to the implementation of a system which pro¬ 
vides reasonable understanding of the more natural utter¬ 
ances of connected speech. 

This system called HEARSAY 1 provides multi-leveled 
knowledge sources ranging from parametric representations 
of the physical features of speech measured every 10ms to 
the conceptual level of common acceptance. Reliable per¬ 
formances for understanding speech are then attained by the 
hypothesize-and-test processes at these various levels where 
these knowledge sources are dynamically related to each 
other through a general structured global data base. 

Concerning images, the University of Massachusetts is 
developing an image understanding system called VI¬ 
SIONS. 2,3 The system transforms visual sensory numeric 
data to a symbolic structure representing a model of the 
image. Conceptual models which are not image-specific are 
stored as a knowledge source in the semantic data base and 
serve to specify the observed model by communicating with 
various levels of features hierarchically extracted through 
spatial windows of various sizes. 

Both of the understanding systems (HEARSAY & VI¬ 
SIONS) are based on the principle of model construction 
(hypothesizing) and the deductive processing (testing) of 
conceptual models by the incorporation of a semantic data 
base (knowledge sources). Thus the associative search of all 
possible conceptual models on a semantic data base plays 
an important role in advanced pattern recognition systems, 


and the efficiency of the search becomes the dominant factor 
in attaining high performance. 

An associative processor ARES proposed by the authors 
before 4 also provides a powerful tool for associative search 
on semantic data bases in general because of the ability to 
associate data items on their deep structures. In the present 
paper, the association principle of ARES which is based on 
coding theory is generalized. The association capability is 
extended by the hierarchical use of a set of different codes. 
Following the overview of the generalized association mech¬ 
anism is an architectural description of ARES which is now 
being developed. By adopting a multi-microprocessor or¬ 
ganization, ARES has gained greater adaptability because 
of the functional flexibility provided by microprogramming. 

In conclusion, the multi-microprocessor ARES provides 
an advanced pattern recognition facility partially bridging 
the gap between human cognitive behavior and logical proc¬ 
essing by computers. 

ADVANCED ASSOCIATION FACILITY 

A semantic data base is defined generally as a collection 
of data items which are symbolic representations of speech 
or image. Data items D' s are coded segmentally or categor¬ 
ically as shown in (1) in such a manner that the relatedness 
of the symbolic partial representations for each segmental 
or categorical block d H is measured in terms of the distance 
of the code vectors associated with them. 

D = (d sl , d s2 , . . . , dsi , . . . , d sp ), 1 <i<p. (1) 

The length of each segmental or categorical block d si is not 
necessarily the same depending upon the application. 

The association mechanism 

Let’s divide D into several blocks of equal length n as 
shown in (2), and apply an error correction scheme of the 
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code of length n with /-error correction capability for each 
block d bj . 

D (d b i , d b 2 , . . ., d b j, , dfoq ), 1 —j—Q • (2) 

The data representation thus derived is denoted as (3). 
e(D) = (e(d bl ), e(d b2 ), . . . ,e(d bj ), . . . , e(d bq )), 

1 (3) 

Each error corrected code vector e(d bj ) represents a blurred 
feature of d bi . The d bj is then called a blurring block, and 
n is normally selected to cover several segmental or cate¬ 
gorical blocks dtf's so as to also blur the influence of arti¬ 
ficial segmentation or categorization of semantic data items. 
Thus the e(D) sustains a global structure of data items which 
are inherently related to each other with equal to or less 
than / distance. 

Now let us apply the content-addressing scheme to 
e(D)'s. When Q is given as a query item, the content-ad¬ 
dressing by e(Q ) is accomplished by checking the number 
of blurring blocks which coincide with each other for all 
e(D)' s in parallel, and then e(D)' s which coincide with e(Q ) 
in equal to or greater than 0(O<0<<?) number of blurring 
blocks are selected. Assuming that both D and e(D) are 
stored as a pair, D' s which are strongly related to Q are 
then specified as associated data items without executing 
the complex calculations of relatedness. 

The space where e(D)' s are accommodated is called a 
blurred data space. Specifying D's through the content-ad¬ 
dressing on e(D)' s in the blurred data space is referred to 
as “association”. The association process is illustrated in 
Figure 1. 

As explained above, the intuitiveness and ambiguity ob¬ 
served in human cognitive behavior are brought into the 
association performance of ARES to some extent by con¬ 
necting the conventional content-addressing and error-cor¬ 
recting technologies, and controlled systematically by the 
length n and the correctable error distance / of the code 
applied. Following is a generalization of the blurring by error 
corrections to reach the more global structure of data items 
before the association. 



error 

correction 
by (n2,k2,t2)-code 


error 

correction 

by(ni,ki,ti)-code n$=ns-i (s- 1 , 2 , — ) 
segmental or categorical ——- conceptual or semantic 

Figure 2—Blurring process 


Let the original data space and blurred data space of level 
5 be denoted as B 0 and B s , respectively. Data items in B 0 
are strongly dependent on physical features directly meas¬ 
urable at B 0 . Data items in B s , however, represent more gen¬ 
eral features commonly possessed by a larger class of data 
items, and the features go up to the conceptual or semantic 
level as the blurring proceeds under the hierarchical appli¬ 
cation of error corrections. 

Perfect codes are desirably selected for the blurring. Fig¬ 
ure 2 illustrates schematically the hierarchical blurring of a 
single block of coded data items for the case of n s = n s - l 
(5=1,2, . . .). Blocks of length rt s located in a circle of radius 
/ g in B s - 1 are merged into one in B s since they must have 
been strongly related to each other. 


Hierarchical process of blurring 


Association based on a deep structured relatedness 


Let («, k, t ) denote a code of length n which has k number 
of information digits with a /-error correction capability, and 
let the blurred data item derived fromD by (n t ,k l ,t 1 )-code 
be represented as D t =e l (D 0 ) where D 0 =Z). Suppose we 
want to have more general structure of data items, then we 
apply (« 2 , k 2 , t 2 )- code to D t by dividing it into the blurring 
blocks of length n 2 . Thus we can proceed the blurring as 
indicated in (4) until we reach the blurredness required for 
the association, where the relation / S s:2/ g _ 1 -l-l should nec¬ 
essarily be satisfied when n^ng-x. 


D 0 —> D x~ e i(D b )— >D 2 — e 2 (Pi)^"' 
*D s — e g (D g _ 1 ) ■>•••. 


(4) 


Content-addressing for the association can be applied at 
any level of blurring when the feature provided is satisfac¬ 
torily general to meet the requirements. Let s* denote the 
blurring level where content-addressing is applied. Then 
D 0 's in B 0 to be associated are directly specified together 
with the related D s 's (0<5<5*) when necessary, without 
tracing back the blurring processes to B 0 since D s (0<5<5*) 
and D s * are stored in a one-to-one correspondence. 

Figure 3 is a schematic diagram explaining how the con- 
tent-addressing and hierarchical blurring are connected for 
performing association. The query item Q 0 (eB 0 ) might in 
some cases be applied externally. The hit marks indicate the 
D s *'s which are content-addressed in B s *. When we have a 
restriction on the number of associated data items, the 
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Associated 




Figure 3—Association through blurring and content-addressing 


threshold d for content-addressing is heuristically selected 
so that hit marks do not exceed a certain limit 8. 

ARES has data modification facility. Suppose Q 0 (eB 0 ) is 
given as a query item. If some D 0 ’s in B 0 are specified by 
Qo through the content-addressing at the blurring level s *, 
these TVs and related D s 's (0<s<,s'*) are replaced by £>o 
and its blurred expressions, respectively. Otherwise, Q s 
(0<5<5*) is registered, unmodified, to B s (0<s<s*). Q 0 
might be given as a weighted average of two data items 
previously stored in B 0 . 

A MULTI-MICROPROCESSOR IMPLEMENTATION 
OF ARES 

In order to effectively carry out the associative search on 
a semantic data base described in the previous sections, the 
following functions should be provided at the computer ar¬ 
chitecture level: 

(1) A hierarchical blurring capability. 

(2) The fundamental content-addressing capability. 

(3) A heuristic control mechanism to obtain a reasonable 
number of associated data items. 

For the hierarchical blurring capability, a control logic is 




Figure 4 —ARES main block diagram 
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necessary which offers an efficient access to the multiple 
blurred data spaces and executes the error correction 
schemes applied for the blurring. Binary cyclic codes are 
useful for a variety of applications because of the flexibility 
of selecting proper length and the easiness of implementing 
the error correction procedure. 

As shown before in Figure 3, the content-addressing is 
accomplished in the blurred data space B s * taking each blur¬ 
ring block of the final stage of blurring as a unit, and for all 
units in parallel. The number of blocks which coincide with 
each other is counted over all the blocks unmasked, and 
compared with a certain predefined threshold value 9. When 
it is equal to or greater than 6, the corresponding hit mark 
is set to 1, and the value of 9 is gradually adjusted by the 
heuristic control mechanism embodied in ARES so that a 
greater number of associated data items as possible are 
obtained within the limit 8. 

In principle, these functions associated with the content¬ 
addressing can be accomplished by the mechanism which is 
incorporated in the uniprocessor ARES originally designed 
for the recognition of handwritten characters. 4 However, in 
order to facilitate the blurring time after time and to effec¬ 
tively create the blurred data spaces, it is extremely desir¬ 
able to provide a logic which works over many sets of blocks 
in data words simultaneously. Furthermore, a blurred data 
space should be spatially distributed again in order to exe¬ 
cute the blurring and content-addressing for all data words 
in parallel. These concepts lead to a conclusion that a mul¬ 
tiprocessor organization is the best of possible approaches 
to attain the advanced association facility described above. 

Architecture of ARES 

Figure 4 shows the block diagram of multiprocessor or¬ 
ganization of ARES. The distributed processing requests the 
Master Control to synchronize the whole logic units. As the 
logic unit, a functionally flexible logic, for instance, a mi¬ 
croprocessor (TI-SN74S481; 16Kbytes memory) is used 
since the data structure of semantic data bases might be 
considerably complicated depending upon the problems to 
be solved. Hereafter, the logic unit is referred to as a cell. 

The Cluster Control controls eight cells all together, and 
the master control governs eight cluster controls. Therefore, 
up to eight different programs may be processed in eight 
different clusters of cells in parallel. On the other hand, in 
the case that all the cluster controls are assigned to process 
the same program, sixty-four parallel operations can be per¬ 
formed in the whole cells. 

The Multiple Response Resolver (MRR) which is a unique 
facility in ARES is made up of a parallel counter with mul¬ 
tiple inputs. Each MRR-1 receives the hit marks from the 
cells attached to it, and the contents of MRR-l’s are col¬ 
lected by MRR-2. The master control compares the content 
of MRR-2 with the predefined value of 8 and adjusts the 
value of 9 when necessary for further content-addressing. 
The design of MRR is basically made clear in Reference 4. 

Figure 5 shows the internal structure of a cell which is the 
nucleus of the association logic. A data item and its blurred 
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representations are accommodated in a cell together with 
the logic encountered in a conventional associative memory. 
The block size and the word size indicate the length of a 
blurring block at each stage of blurring and the length of a 
data item, respectively. 

We may conclude that the clustering of cells supplies high 
reliability and functional flexibility; the loss of a single cell 
function or MRR-1 results in a slight decrease of processing 
power but no complete breakdown of the system, and each 
cluster of cells can perform its own function individually. 

Functional behavior of ARES 

The functional behavior of ARES for manipulating a se¬ 
mantic data base is explained as follows: The mini-computer 
HP-21MX is used as a Host Computer, and recognizes 
ARES as an associative memory with extremely high intel¬ 
ligence. The operation is initiated when the host computer 
issues a global command to the master control. The master 
control analyzes the command and gives the related sub¬ 
commands to the cluster controls. This implies the function 
of pointing out the start addresses of the microprograms to 
be performed, which are previously stored in the cluster 
controls. Then each cluster control carries out the specified 
operation individually. 

Figure 6 shows the relationship between the blurred data 
spaces and the corresponding cells in the blurring mode. 



Each blurred data space is divided into subspaces and data 
items contained in a space are distributed to the cell mem¬ 
ories so that they can be processed in parallel. The error 
correction for the blurring is accomplished through ALU 
shown in Figure 5, and controlled by the microprograms in 
the cluster control. 

Figure 7 indicates the same relationship in the content¬ 
addressing mode. A query item and the value of 8 are sent 
to each cell in parallel by the master control. The content¬ 
addressing is then performed in the blurred data space B s * 
which are distributed in cell memories, with 6 heuristically 
controlled by MRR under the parameter 8. The control of 
d is implemented in hardware because the very fast multiple 
associations are required repeatedly. The hardware is de¬ 
signed in consideration of the trade-off between the func¬ 
tional requirements imposed and the cost of implementation. 


CONCLUSION 

In this paper, the association principle presented before 4 is 
extensively developed in order to be applied to the associ¬ 
ative search on semantic data bases for speech or image 
understanding. Data items are coded in such a manner that 
the relatedness of the symbolic partial representations of 
segmental or categorical blocks of data items is measured in 
terms of the distance of the code vectors associated with 
them, and they are blurred by the hierarchical application 
of error correction schemes. Thus, by simply applying con¬ 
ventional content-addressing schemes to the blurred data 
items, the associative search of original data items is accom¬ 
plished on their deep structured relatedness without exe¬ 
cuting the complex calculations of relatedness on original 
data representation. 

Following the overview of the generalized association 
mechanism, we presented an architectural description of 
ARES which is now being developed. From the viewpoint 
of computer architecture, ARES has the following charac¬ 
teristic features: A multi-microprocessor organization is 
adopted using HP-21MX mini-computer and TI-SN74S481 
Schottky I 2 L bipolar microprocessors as a host computer 
and cell components, respectively. This provides both the 
structural extendibility and functional flexibility supported 
by microprogramming to adapt to a variety of applications 
in the field of artificial intelligence. Furthermore, ARES 
facilitates heuristic control of the number of associated data 
items, which is useful to solve pattern recognition problems. 

So far we have discussed the associative search on se¬ 
mantic data bases for speech or image understanding. How¬ 
ever, ARES can be well applied to natural language proc¬ 
essing. For example, the associative-categorical model 
proposed by Haralick and Ripken 5 for semantic word re¬ 
trieving is dealt with by means of representing categorical 
relatedness of data items by code distances. 6 

In conclusion, a multi-microprocessor ARES provides an 
advanced pattern recognition facility bridging the gap be¬ 
tween human cognitive behavior and the logical processing 
by computers. 
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The STARAN architecture and its application to image 
processing and pattern recognition algorithms 

by J. L. POTTER 

Goodyear Aerospace Corporation 
Akron, Ohio 


INTRODUCTION 

The STARAN E is a general purpose parallel computer. 
However, certain aspects of the STARAN E’s architecture, 
specifically, a single instruction stream-multiple data 
stream organization, high speed I/O, a flip (permutation) 
network and a conditional operation capability on each par¬ 
allel processing element, are particularly pertinent to the 
areas of image processing and pattern recognition. 

Some of the characteristics of image processing and pat¬ 
tern recognition which provide a good fit to the STARAN’s 
architecture are repetitiveness, spatial dependencies, com¬ 
plex parallel decision making and high I/O to computation 
ratios. These characteristics and their corresponding 
STARAN E architectural accommodations are described in 
detail. The other aspects of STARAN’s architecture are 
described in only enough detail to provide a basis for dis¬ 
cussion.* 


BACKGROUND 

The STARAN E consists of an associative processor (AP) 
control module and a number (1 to 32) of associative arrays. 
The AP module consists of the AP control circuitry itself 
and bulk core. Figure 1 shows the basic STARAN architec¬ 
ture. 

The arrays can be thought of as consisting of high speed 
memory, low speed memory and a bank of processing ele¬ 
ments. Each array is organized into 256 “words.” Associ¬ 
ated with each “word” is a processing element (PE). Each 
word is 9K bits long with 1024 bits of high speed memory 
and 8192 bits of lower speed memory. Figure 2 illustrates 
the conceptual array organization and a general purpose 
layout for a 512x512 8 bit/pixel image. 

Arithmetic operations are performed in parallel on every 
(enabled) word of memory, one bit at a time. That is, the 
least significant bit of every word in a field (a bit column 
vector) is added (multiplied, etc.) to the corresponding least 
significant bit of every word in a second field and stored in 

* For a detailed description of the STARAN line of computers see References 
1, 2 and 3. 


the least significant bits of the sum (product, etc.) field. The 
carry bits are saved and added into the second bit slices of 
the arguments. This process is repeated until the entire fields 
have been processed. 

ARCHITECTURE—ALGORITHM PARINGS 

Certain characteristics of image processing and pattern 
recognition mesh perfectly with some of the architectural 
features of the STARAN E. In the following paragraphs. 



Figure 1—Basic STARAN architecture 
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some characteristics of image processing and pattern rec¬ 
ognition algorithms will be identified and described. Then 
the corresponding architectural feature of the STARAN E 
will be discussed and shown how it facilitates implementa¬ 
tion of the algorithms. 

REPETITIVENESS 

Images consist of large volumes of data. A standard TV 
monitor size image contains over a quarter of a million pixels 
(picture elements). The images obtained from satellites may 
contain many millions of pixels. Almost all image processing 
and pattern recognition algorithms consist of performing the 
same sequence of operations for every pixel in an image. 
This aspect of image processing fits perfectly with the single 
instruction stream-multiple data stream organization of the 
STARAN. 

The associative processor (AP) control portion of the 
STARAN computer provides a single sequential instruction 
stream to the associative arrays. Each associative array 
contains 256 processing elements (PE’s) and a fully equipped 
STARAN E may have up to 32 arrays. Thus the single 
instruction stream can control from 256 to 8192 PE’s result¬ 
ing in a processing capability of from 11 to 356 million 32- 
bit adds per second (MIPS). Figure 1 illustrates the single 
instruction stream-multiple data stream organization of 
STARAN. 

“Inherently serial” algorithms such as classical Maximum 


Likelihood classification are easily implemented in parallel 
with the above organization. This is because the algorithm 
is still executed in serial for every pixel, but from 256 to 
8192 pixels can be handled with one pass of the algorithm. 
With the large number of pixels that need to be processed 
in a typical image, parallel application of the algorithm is the 
only practical answer. 
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Figure 3—Flip network 
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Figure 5—Butterfly diagram for FFT 


SPATIAL DEPENDENCIES 

Image data is inherently two dimensional and processing 
algorithms often must deal with this “built in” organization. 
In general, there are two different types of relationships. 
The neighborhood relationship requires that pixel neighbors 
in the two dimensional plane be readily accessible. The 
second type of relationship is normally spatially more ex¬ 
tensive and is typified by the Fourier Transform where the 
pixel and its neighbors at power of two intervals for an 
entire row or column are related. 

The ST ARAN associative arrays are well suited for ac¬ 
commodating both types of dependencies. Each array, as 
shown in Figure 3, has a flip network located between the 
memory and PE portions. This network performs as a spe¬ 
cialized shift register and provides great flexibility in ac¬ 
cessing pixels which are spatially related. 

In particular, the flip network can accomplish power of 
two rotates within power of two sized fields with no time 
penalty. That is, a 1 bit rotate in 128 2 bit fields; 1 and 2 bit 
rotates in 64 4 bit fields; 1, 2 and 4 bit rotates in 32 8 bit 
fields; on up to 1, 2, 4, 8, 16, 32, 64 and 128 bit rotates in 
one 256 bit field. (See Figure 4) This means that inter-word 
operations at these intervals can be performed in parallel at 
the same rate as intra-word operations. Moreover, all inter¬ 
word operations can be performed with only a slight penalty. 

The flip network then provides a very efficient method of 
implementing algorithms such as the Fast Fourier transform 
which utilize the spatial power of two interrelationships be¬ 


tween pixels. This power of two spatial relationships is fre¬ 
quently expressed in the butterfly diagram shown in Figure 
5. A detailed discussion of how the FFT can be implemented 
in the STARAN in log N steps of 1 add, 1 subtract, 2 real 
multiples and 2 exchanges has been published elsewhere. 4 

Template matching and spatial convolution are two algo¬ 
rithms which require the neighborhood type of pixel access. 
These processes can be implemented with little or no time 
penalty for the required spatial relationships. In particular, 
2x2, 3x3, 5x5, 9x9 and 17x17 displacements require no 
time penalty. Other displacement amounts up to 16x16 
require at most one extra shift except for 12x 12 and 14x 14 
which require two shifts. In general, the required shifting 
for processing any window or template size within the 256 
word array (i.e. 255x255 or less) is insignificant in overall 
algorithm time. 

PARALLEL DECISION MAKING 

An important aspect of image processing and pattern rec¬ 
ognition is decision making. Many algorithms perform dif¬ 
ferent operations as a function of the data. The complexity 
of the process is typified best perhaps by scene analysis 
techniques. In these situations it is essential to be able to 
record the exact state of each individual datum in the image. 

Each STARAN array contains a special register which 
can be set as the result of searches (LE, GT, etc.) and 
arithmetic operations. This mask (or M) register can be used 
to select a subset of the words in an array to participate in 
subsequent operations. The results of tests, conditions and 
states can be stored, retrieved and operated on to achieve 
any desired logical combination of tests. Figure 6 indicates 
the physical location of this register. 

Two examples of how the M register can be used are 
template matching and hierarchical structuring. To start a 
3x3 template match, the M registers are set to all ones so 
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Figure 6—M register configuration 
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Figure 8—“Typical” scene analysis configuration 


that every word participates in the search. The first value 
of the template is loaded into the common register and the 
first columns of all the arrays are matched against it. Every 
successful match is recorded in the Y register. The Y register 
is shifted down one and moved to the M register, the second 
template value is moved to the C register and the column is 
searched for a match of the second template value but on 
only those words with a corresponding one in the M register 
(i.e., only those which successfully matched the first value). 
Consequently at the end of the second search the Y register 
contains a one for every word and only those words which 
successfully matched both the first and second template 
values (the state shown in the M and Y registers in Figure 
7). The procedure is repeated for each of the template values 
with an upward shift and column increment after searching 
for the third and sixth values. At the end of this process, 
the M register of all enabled arrays will contain a one (shifted 
down by two) corresponding to every successful template 
match covering columns 1, 2 and 3. The process is repeated 
for every set of columns to be searched. 

Note that the columns need not be contiguous to be 
searched. Thus two situations can be easily handled. First, 
the columns can be organized in an order which facilitates 
another algorithm and/or input/output situations. Second, the 
template need not be contiguous but may cover essentially 
any size area in any desired manner. Both of these capabil¬ 
ities emphasize the flexibility of data processing in the 
ST ARAN arrays. 

In a scene analysis application, each pixel might have a 
set of flags and auxiliary fields associated with it as shown 
in Figure 8. Then searches of the type “obtain the edges of 
object 5 on level 2” would be easily implemented in parallel 
by: first, searching all pixels in a column for object 5, then 
searching the matched words for level number 2 and then 
ANDing the edge flags with the Y register. The result is the 
answer to the search for the given pixel column. The process 
would then be repeated for each column under considera¬ 
tion. This example illustrates how easy it is to save the 
result of previous algorithms as flag vectors and codes in 
parallel and that this information is readily available for 
subsequent processing and analysis. 

INPUT/OUTPUT VERSUS COMPUTATION 

It has been established that in general, image processing 
involves large volumes of data. The algorithms vary quite 
markedly however in the degree of computation involved. 
Simple grayscale remapping is such a useful function that 


special hardware circuitry is often contained within the dis¬ 
play devices. In order to make such changes permanent, 
however, a computer must be used. Operations such as 
changing (remapping) the grayscale values of images require 
only one operation per pixel but must be performed on every 
pixel. These algorithms represent the high I/O-low com¬ 
putation end of the spectrum. At the other end are such 
algorithms as the domain transforms. Frequently these pro¬ 
cedures require numerous arithmetic operations on a per 
pixel basis. For example, a two dimensional Fast Fourier 
Transform for a 512x512 pixel image requires 54 multiples 
and 90 adds per pixel (for a serial computer). 

The STARAN has three I/O paths. The common register 
path (shown in Figure 1 and at the top in Figure 9) operates 
at between 12-15 million bits per second (MBPS). It is most 
useful for “broadcasting” data such as constants and param¬ 
eters to all arrays in parallel. It is a 32 bit wide path. 

The most useful path for data I/O is the 32 bit wide mul¬ 
tiplexed I/O bus into and out of each array. This bus is 
capable of operating at between 80 and 640 MBPS. Thus an 
entire 512x512 8 bit per pixel image can be input or output 
in from 26.2 to 3.3 milliseconds. This data path is connected 
to a crossbar switch so that it can be used to transmit data 
between arrays as well as to peripheral storage or image 
display devices. 

The fastest bus is the 256 bit parallel I/O bus which can 
operate from 512 to 2560 MBPS. The data transfer rate on 
this bus is such that special peripheral configurations are 
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required. Consequently, it is best suited for special purpose 
applications. 

Every array module is capable of being connected to 
either of two controllers. Thus in those applications where 
the equivalent of double buffering is desirable, some of the 
arrays can be switched to an I/O controller while the re¬ 
mainder are used for computation. 

SUMMARY 

The multiple data stream parallelism of the STARAN allows 
it to perform a “serial” algorithm on up to 8192 data ele¬ 
ments simultaneously. This is important in image processing 
where large volumes of data are processed. Images are in¬ 
herently two dimensional, but the large array size of the 
STARAN E, readily allows an entire 512x512 8 bit/pixel 
image to be stored in one array with essentially its two 
dimensional topography intact. The array addressing struc¬ 
ture and the array flip network provides easy access to every 
pixel and its neighbors in a simple efficient manner. The 
mask (M) register operation in an array enables complex 
decision processes to be made on any subset of pixels in an 
array. All of the above aspects of the STARAN’s architec¬ 
ture would not be valuable if it could not be efficiently used. 
The three I/O paths into every array provide the capability 
to efficiently broadcast parameters and constants as well as 
loading and unloading image data in an expeditious manner. 
Thus it is apparent that many of the architectural features 


TABLE 1.—Sample Algorithm Processing Times 


Function 

Description 

Image Size 

Speed* 

Magnification 

2.5 x cubic 
convolution 
interpolation 

512X512 

8 bit/pixel 

588 milliseconds 

Convolution 

3x3 window 

512X512 

8 bit/pixel 

700 milliseconds 

FFT 

1 Dimension 

512 16 
bit/pixel 

2.7 milliseconds 


* Measured times in the STARAN B machine exclusive of I/O. The STARAN 
E instruction execution time is approximately 20 percent faster. 


of STARAN are ideally united for image processing and 

pattern recognition. This conclusion is confirmed by the 

execution times shown in Table I. 
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The criterion COBOL system 
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INTRODUCTION 

Higher level programming languages provide problem sol¬ 
vers an access to computers without requiring them to be¬ 
come computer experts. In the traditional environment, 
compilers perform the relatively complex translation from 
the higher level language statements into the machine in¬ 
structions. The differences between the complex data struc¬ 
tures and operations of languages and typical simple struc¬ 
tures of machines makes this a wide chasm to bridge. 

The mismatch between data and operations of the lan¬ 
guages and machines appears clearly when one considers a 
language such as Cobol, APL, PL/I or Snobol4. In each of 
these, the fundamental operations and even basic data types 
do not translate easily because the language design depends 
on no one machine. The implementor must build mecha¬ 
nisms ranging from full software interpreters to elaborate 
subroutine libraries to perform the higher level language 
operations on higher level language data. 

Despite the costs, the payoffs of using higher level lan¬ 
guages make them worthwhile. And with the widespread 
use of microprogrammable processors as the base of com¬ 
puting systems, designers can realize the once primarily 
theoretical higher level language machines. 1 An interpreter 
for the operations of a higher level language, written in 
firmware rather than software becomes a higher level lan¬ 
guage machine. 

The NCR Criterion Cobol System implements an indirect 
higher level language machine for 1974 ANSI Cobol in a 
conventional microprogrammed environment, permitting 
execution of a single run unit composed of both Cobol and 
other language modules in a highly efficient manner. 

The system consists of a microprogrammed hardware pro¬ 
cessor supporting both conventional and Cobol machine 
firmware in a multiple virtual machine environment, a mul¬ 
tiprogramming virtual resources operating system, a 1974 
ANSI Cobol compiler designed to run in a virtual storage 
environment, the necessary execution time support subsys¬ 
tems and routines, and Cobol symbolic debug and runtime 
tuning subsystems. 

The discussion focuses on the characteristics of the sys¬ 
tem as a Cobol environment and the structure of the Cobol 
virtual machine, the compiler, and the support subsystems. 


THE CRITERION SYSTEM 

NCR system architects chose a higher level language ap¬ 
proach as part of the basic design of the Criterion, a new 
family of computers to replace the aging Century line. The 
other part of the design fully supports, without modification, 
existing NCR Century application programs, no matter what 
their programming language. 

Cobol, probably the most widely used data processing 
applications language and hence of most interest to the com¬ 
pany, serves as the base of the higher level language system. 
The new virtual resources executive (VRX) operating sys¬ 
tem design includes support of multiple virtual machines. 
Cobol Virtual Machine (CVM) firmware, a new American 
National Standard Cobol 2 compiler, and a set of Cobol ex¬ 
ecution time support subsystems form the resulting Criterion 
Cobol System (CCS). 

A very fast (56 nsec) pipelined microprogrammable pro¬ 
cessing element forms the base hardware of the Criterion 
system. Three storage units attach to it: register storage 
(RSU), instruction storage (ISU) and memory storage 
(MSU). The RSU holds temporary work areas for the ele¬ 
ment. Microcode executes primarily out of the ISU, al¬ 
though some low-speed instructions execute from the MSU. 
The MSU areas not used by firmware become the main 
storage used by the operating system and application pro¬ 
grams. 

The hardware supports a paged memory environment with 
a maximum addressing capability of 16 megabytes (MB). 
The processing element performs address translations with 
interrupts to the executive on page faults. 

The customer can receive either of two sets of firmware 
and software for the Criterion: the real resources system 
(RSI) or the virtual resources system (VS1). RSI emulates 
NCR Century computers and permits use of the Criterion 
as a direct replacement. VS1 supports the virtual resources, 
including virtual storage and the multiple virtual machines. 
The two current virtual machines provide an extended ver¬ 
sion of the Century instruction set and architecture, sup¬ 
porting the NEATVS assembly language (NVM) and the 
new CVM. 

While these two machines co-reside in the system and 
support many similar elementary data types, they have sev- 
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eral fundamental differences in design orientation. NVM 
uses binary data extensively while CVM works best with 
decimal data. NVM uses an absolute addressing scheme, 
but with addressing a maximum of 32 kilobytes (KB) offset 
from a register, except in the first 64 KB. CVM uses bases, 
but allows offsets up the full virtual address space limit. 

Both NVM and CVM share a common stack-oriented 
calling mechanism and a common virtual addressing scheme 
so that a program executing in either machine mode can 
invoke a module in the other. Each entry point to a module 
has an entry point control item (EPCI) indicating the virtual 
machine mode for execution of that module. Machine state 
can change on any call. 

COBOL VIRTUAL MACHINE 

The Cobol Virtual Machine (CVM) implements an indirect 
higher level language machine for the Cobol data types and 
structures and for a large subset of the Cobol operations. 
An object module generated for execution by the CVM 
contains three principal memory areas and some minor aux¬ 
iliaries. 

The data area contains all the data described in the pro¬ 
gram either explicitly or implicitly. Implicit data includes 
various file tables, collating sequence tables, debugging ta¬ 
bles, and literals. Data definition entries in the Data Division 
describe explicit data. 

The descriptor area contains an internal representation of 
the information contained in the data description entries. 
Each referenced data element has a descriptor. Each de¬ 
scriptor contains location, size, type, index, and editing 
information for the data element. 

The code area contains the translated procedure division 
statements. The basic internal operations (for example, sim¬ 
ple MOVE or ADD) translate to single machine instructions. 
More complicated statements, such as those having multiple 
receiving fields, decompose into simple Cobol statements 
for code generation. With operation codes generally type¬ 
less, automatic conversion and editing occur when required 
by the referenced descriptors. 

Data types 

The fundamental elementary data types in the CVM cor¬ 
respond to the variations on COMPUTATIONAL and DIS¬ 
PLAY types in Cobol. 

Packed decimal items represent computational data inter¬ 
nally. The requirement for frequent conversions makes 
packed decimal the most efficient format to use in CVM. 
Depending on the Cobol picture, the item may or may not 
have a sign. Computational data items may vary in precision 
from 1 to 18 digits and may have an implied decimal point 
at any position adjacent to a digit or may have scaling to a 
maximum of 18 positions from the farthest digit. An addi¬ 
tional four-bit trailing half-byte represents the sign, when 
present. 

Both numeric and nonnumeric data items occur. Numeric 
display items follow the same rules for computational items 
on precision and decimal point position. ASCII characters 


‘0’ through ‘9’ represent the digits. If the programmer spec¬ 
ifies a signed item with a “separate sign,” an ASCII * + ’ or 

’ represents the sign. The user specifies whether the sign 
goes before or after the digits. If the user requests a “zone 
sign,” the sign indicator combines with the first or last digit 
to form an ASCII character according to this table: 

no sign 0123456789 
+ sign {ABCDEFGH I 
- sign } J K L M N O P Q R 

Strings of ASCII characters form the nonnumeric data 
items, having the internal categories of alphanumeric, al¬ 
phanumeric justified right, numeric edited, and alphanu¬ 
meric edited, and the properties specified in the Cobol stand¬ 
ard. 

The CVM provides two additional data types. Signed bi¬ 
nary (two, four, or eight bytes) provides data communica¬ 
tions with the NVM. User documentation recommends 
against its use for other purposes. Special index-handling 
commands manipulate index data items for the table han¬ 
dling operations. The internal structure of these items re¬ 
mains transparent to the user. 

Descriptors 

The descriptor area may contain any combinations of four 
types of descriptors. Two of the descriptors, used for com¬ 
pact data descriptions, require four bytes each. Only certain 
elementary items use them. The other two descriptor types 
define the more complicated data structures. The long de¬ 
scriptors take eight bytes each. 

Short nonnumeric descriptors describe alphanumeric or 
alphanumeric justified right data items whose length does 
not exceed 4095 characters. Short numeric descriptors de¬ 
scribe decimal, packed decimal, or binary items. In both 
cases, the item must begin in the first 32768 bytes of the 
data area. 

Long nonnumeric descriptors describe alphanumeric, al¬ 
phanumeric justified right, or alphanumeric edited data 
items. They allow a maximum length of 65535 characters. 
Other fields describe the number of dimensions the item 
has, whether an edit information mask pointer follows, and 
the method of addressing used for the data. Long numeric 
descriptors vary from the long nonnumeric descriptors in 
that they have decimal position and length fields in place of 
character length fields. 

Edited data items require edit information. An edit mask 
provides the description of the format of the result. The 
mask derives from the picture specified in the source pro¬ 
gram. For alphanumeric edited and numeric edited items, a 
flag in the descriptor indicates an extension to twelve bytes, 
with the additional four bytes pointing to the edit mask and 
containing the numeric length for numeric edited items. 

Addressing 

Addressing in the CVM uses the standard 24-bit Criterion 
virtual address mode. Within a program, the system com- 
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putes the effective virtual address as an offset from the start 
of the data, descriptor, or code area. A linkage table contains 
the three bases. On entry to a module, the XLINK register 
of the machine points to a table of bases. The first three 
entries contain bases for data, code, and descriptor areas. 
An additional eight entries point to various other control 
and debugging locations. 

Effective data addresses for descriptored items come as 
direct offsets for the data area or compute from addresses 
passed through parameters. Automatic computation of in¬ 
dices for tables occurs when table references appear. 


Code 

The CVM instructions, as indicated earlier, closely match 
Cobol statements for their operations. The common state¬ 
ment options form part of the instructions, where appropri¬ 
ate: numeric operations have rounded and size-error flags. 

Each operation contains a bit indicating whether it begins 
a paragraph. The debugging system, described later, uses 
this information. 

A few of the typical commands demonstrate the nature of 
the CVM instruction set: 

MOVE commands act like simple move statements in 
Cobol, reformatting data as necessary from the source 
to the destination. A single instruction provides for all 
conversions and editing. The MOVE-BYTE commands 
allow movement of a single character literal. 

COMPARE commands form the basic operations of the 
IF statements. The principal COMPARE command 
handles both numeric and nonnumeric data. Separate 
instructions provide IF-ALPHABETIC and IF-NU- 
MERIC tests. Comparison tests use either native ASCII 
collating sequence or one derived from some table, such 
as EBCDIC. The user specifies the choice with a PRO¬ 
GRAM COLLATING SEQUENCE clause. 

GOTO commands provide for internal branching, with 
Cobol GO TO and GO TO . . . DEPENDING state¬ 
ments as basic instructions. Others include a conditional 
GO TO for size error detection and an indirect GO TO 
for PERFORM handling. 

Arithmetic commands mimic the Cobol arithmetic state¬ 
ments (except the COMPUTE statement). The com¬ 
mands require numeric source operands and numeric or 
numeric edited result items. Both two- and three-ad¬ 
dress operations occur, corresponding to statements of 
both forms: 

ADD A TO B. 

ADD A, B GIVING C. 

Programmers request rounding and size error detection 
in the arithmetic commands by clauses in source: 

ADD A, B GIVING C ROUNDED; 

ON SIZE ERROR . . . 


CALL commands invoke a standard Criterion calling se¬ 
quence. The program sets up parameters in the standard 
parameter stack using a MOVE-VMA command to 
compute the virtual address. A state switch to NVM 
may occur during the call. 

EXIT commands provide the Cobol EXIT PROGRAM 
statement for a called subprogram. On exit to the calling 
program, a state switch may occur to return to the 
NVM state if an NVM program invoked the CVM pro¬ 
gram. 


THE COBOL COMPILER 

The Cobol compiler (CBL) for the CCS converts 1974 
ANSI Cobol programs to CVM object modules. Its logic 
design generally matches that of a typical textbook example 
Cobol compiler, but with the unique feature of its extensive 
use of the Criterion virtual address space. 

The implementation team used contemporary software 
engineering techniques in constructing the compiler. They 
wrote the bulk in a subset of the NCR Software Writer’s 
Language (SWL). NEATVS assembly language functions 
provide a few utility services, and Cobol coded modules 
(after bootstrapping) do the input-output. CBL contains 
more than 200 modules, linked together in a tree-structured 
manner. 

The development process, which occurred concurrently 
with the development of the Criterion and the operating 
system, heavily used a variety of tools. A commercial time¬ 
sharing system served as the initial development engine. An 
earlier Cobol development project formed a base for about 
30 percent of the code. Snobol4 encoded programs trans¬ 
lated this code to the SWL syntax required by CBL. The 
compiler team coded the remainder of the compiler in the 
SWL subset. A tools group developed a compiler for the 
time-sharing system to match the subset of SWL used by 
CBL. The resulting object program for CBL executed under 
a simulated virtual storage test environment developed for 
the CBL project. The team debugged most of the compiler 
logic before the new hardware and operating system became 
available. 

When the Criterion system could run with a rudimentary 
operating system, a modified remote entry system transmit¬ 
ted source modules to the NCR equipment for compilation 
and testing. This linkage remained in place throughout the 
development cycle. The master library remained on the 
time-sharing system until the Criterion system became fully 
stable. 

User’s view of CBL 

To the Cobol programmer, CBL appears as an easy-to- 
use conventional compiler. Information provided at compi¬ 
lation time relates the execution time debugging output to 
the source program. The compiler can produce extensive 
diagnostics in clear English. The programmer specifies a 




1052 


National Computer Conference, 1978 


requested minimum level, from non-ANSI feature through 
serious errors only. Diagnostics refer to the source program 
by source line and column number. 

Input to the compiler comes from source cards, from a 
Cobol source library, or (for compatibility with earlier sys¬ 
tems) from NCR “SPUR” format files. Outputs include the 
source listing and a diagnostic summary, along with the 
object module on disc, if the compiler generates it. The user 
may specify optional cross-reference, object listing, and ob¬ 
ject map printed output. The Cobol library facility furnishes 
texts for the Cobol COPY statement. 

Object modules produced by the compiler contain the 
code, data, and descriptor tables required by the CVM, the 
control tables used by the link editor, and optionally, de¬ 
bugging tables used by the execution time Cobol symbolic 
debugging and tuning systems. 

Tables and texts 

The structure of internal tables and texts gives CBL the 
properties which make it particularly well-suited for virtual 
storage operation. As used in this discussion, a table struc¬ 
ture holds randomly accessed data and a text structure 
stores sequentially accessed data. 

Tables in CBL contain the information usually associated 
with the compilation process: data names and attributes, 
procedure (label) names and attributes, file properties, lit¬ 
erals, etc. In keeping with the rules for virtual storage pro¬ 
gramming, the design of tables makes them compact. A 
vector of data records forms a table. If a record requires 
eight or fewer bytes, the table consists of the collection of 
records. If the record size exceeds eight bytes, the table 
contains a vector of three-byte record pointers and a collec¬ 
tion of records allocated, as required, from a storage pool. 
This approach permits CBL to compile the widest mix of 
Cobol programs, keeping the table storage as compact as 
possible. 

A few principal internal tables hold most of the data 
passed between phases: 

DATA TABLE. The data table contains the data name 
information and the attributes of the data name. The 
attributes include both external properties, such as line 
and column where defined, and internal properties, such 
as data type or level in structure. An auxiliary table 
entry holds less frequently used items which cannot fit 
into the data name attribute table. 

PROCEDURE TABLE. The procedure table contains the 
procedure name (Cobol paragraph and section label) 
information. It uses the auxiliary table when necessary. 

PICTURE AND LITERAL TABLE. The picture and lit¬ 
eral table contains pictures and literals which appear in 
the source program, literal generated by the compiler, 
and edit masks generated from the pictures for numeric 
edited and alphanumeric edited data items. 

FILE TABLE. The file table takes the information de¬ 


veloped by the compiler from environment, data, and 
procedure division references to files. The file table 
links to the data table and to the auxiliary table to store 
additional information. 

Texts in CBL store the information frequently associated 
with temporary scratch files. A text appears as a large vector 
of relatively small records. The compiler places entries into 
a text or retrieves entries from a text in a sequential manner. 
In this way the virtual storage paging mechanism moves the 
idle information to the page storage device and brings it 
back when referenced. The overhead associated with normal 
file input-output operations disappears and, for a small 
Cobol program in a relatively idle system, the text pages 
may never need paging out. 

The principal internal texts used for passing information 
between phases in CBL are: 

COBOL TEXT. The Cobol text (CTEXT) contains a 
regularized representation of the original source pro¬ 
gram with data names, procedure names, and reserved 
words, pictures, and literals represented by entry num¬ 
bers in appropriate tables. 

GENERATOR TEXT. The generator text (GTEXT) con¬ 
tains an intermediate form of the Cobol program, with 
references to table entries for data items, labels, literals 
and other items. The GTEXT contains a series of 
“packets” which correspond roughly to assembly lan¬ 
guage statements in a compiler which compiles to as¬ 
sembly language in the “front end.” 

OBJECT TEXT. The object text (OTEXT) contains the 
object information produced by the object generator, 
but before packaging into an external object module 
format. 

ERROR TEXT. The error text (ETEXT) contains the 
information produced by the compiler analysis phases 
and used by the diagnostic formatter. 


Compiler phases 

CBL divides into from five to ten phases, depending on 
how one wishes to number them. If one chooses the mini¬ 
mum number, several of the phases divide into subphases. 

A main driver initializes compiler work areas, opens 
source and listing files, processes user request options and 
then calls the first compiler phase. It proceeds through each 
compilation phase, checking for a possible compilation 
abandonment after each, until it reaches the object genera¬ 
tion phase. It invokes generation only if: 

a. the user has not suppressed it (an option), and 

b. no serious error has suppressed it. 

After object generation, the object formatter outputs the 
code, the main driver prints a compilation summary, closes 
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the files, and returns to the monitor to terminate compila¬ 
tion. 

The source scanner, first phase of the compiler, converts 
the Cobol source lines read from input to the CTEXT. A 
hashing mechanism (buckets and chained entries) builds the 
symbol table through the data and procedure tables. Pic¬ 
tures, literals, and unknown symbols go into the picture and 
literal table. The scanner produces the source listing and 
places syntax diagnostics into the ETEXT. 

The source scanner does all the Identification Division 
processing, principally recording the program name. 

Three subphases handle the Environment and Data Divi¬ 
sion processing. One of these, the picture analyzer, scans 
the picture strings for meaning and errors, and sets up at¬ 
tributes for data associated with the picture. It also builds 
edit masks corresponding to specifications in the picture. 
Two subphases extract source from the CTEXT and parse 
and analyze other table information in the divisions, building 
and modifying the appropriate table entries. All three sub¬ 
phases can generate diagnostic requests into the ETEXT. 

The Procedure Division processor takes the source text 
from the CTEXT and information from the tables and builds 
the GTEXT. It also makes entries, if necessary, in the 
ETEXT and other tables. 

The final front-end phase, the literal pooler, moves entries 
from the picture and literal table into the literal pool, a small 
text. 

Two back-end subphases generate the object module. An 
object generator analyzes the tables and literal pool to build 
the data and descriptor areas. It uses the results from these 
and the GTEXT to construct the code area, and also pro¬ 
duces the required link editor tables. All of this information 
goes into the OTEXT, which passes to the object formatter. 
The object formatter writes the object module on disc and 
optionally produces an object map and listing. 

The main program has two utility phases available when 
it needs them. A cross-referencer produces an alphabetic 
reference listing of all data and procedure names. The main 
program runs this procedure, on user request, immediately 
after the source scanner. The diagnostic formatter runs just 
before the end of compilation if any phase has emitted a 
diagnostic request to the ETEXT. 


EXECUTION TIME SUPPORT 

Criterion Cobol System execution time support includes 
a set of “global” routines for handling the operations re¬ 
quired for input-output, sort-merge, communications, and 
operating system interface statements. It also contains soft¬ 
ware/firmware debugging and tuning subsystems, described 
in the next section. 

Global routines in VRX, residing in a protected memory 
area accessible to all simultaneously executing programs, 
consist of reentrant code. 

The input-output system, the Criterion Access Method 
(CAM), provides Cobol support for sequential, relative, and 
indexed files. Relative and indexed files may reside on any 


disc type supported on the machine. Other devices handle 
sequential files only. 

The sort-merge system provides the Cobol support for 
sorting and merging magnetic tape and disc files. 

The Message Control System (MCS) provides functions 
for the Cobol Communications functional module. The Net¬ 
work Description Language (NDL) processor sets up the 
communications paths according to user specifications. 

An operating system interface module, invoked primarily 
for Cobol ACCEPT and DISPLAY statements, supports 
such actions as reading the clock or calendar, communicat¬ 
ing with the operator, and reading control card parameter 
information. 


COBOL SYMBOLIC DEBUG AND TUNING AIDS 

The Cobol symbolic debug system (COBUG) provides the 
Cobol programmer with a variety of information for sym¬ 
bolically monitoring the program and data flow in an exe¬ 
cuting program. 

The first instruction of each paragraph contains a special 
flag bit. Before executing the instruction, the firmware 
checks that bit and an internal symbolic debug switch. With 
both conditions on, the COBUG subsystem takes control. 
It displays information requested by the user. The user, 
through control statements, may request a variety of types 
of information: 

DUMP of all or selective data items at specific places in 
the program. Options include specification of which 
time during execution the dump should occur, the num¬ 
ber of times, and relational conditions: 

DUMP ABC AT XYZ FOR 10 TIMES WHEN A 

IS LESS THAN B 

TRACE of all or selective paragraph flow. Options 
include ranges, and the number of times: 

TRACE XYZ TO PDQ ON 5 FOR 12 TIMES 

FLOW of program execution. If requested, the firmware 
records the execution of each instruction. At program 
termination time, the COBUG system formats and 
prints a listing of the data. The user, who must under¬ 
stand the CVM operation, can use this listing to detect 
problem points in the program. 

The Cobol analysis routine for runtime tuning (CAR- 
TUNE) provides the Cobol programmer with a mechanism 
for collecting execution time statistics. The Cobol compiler, 
on request, builds a source line table and three activity 
tables in the object module. At execution time, the CVM 
firmware optionally maintains tables of the number of times 
each type of operation code executes, the number of refer¬ 
ences to each data item, and the number of times each 
instruction executes. 

At the end of execution, CARTUNE gathers the infor¬ 
mation and prints an analysis. The user may request specific 
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reports: source line executions, data references, and object 
instruction execution history. These data assist the program¬ 
mer in locating the inefficient portions of the program for 
improvement. 

THE RESULT 

The Criterion Cobol System provides the Cobol user with 
an integrated system optimized for highly efficient Cobol 
program development and execution. 
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Review of the CLIP image processing system 


by M. J. B. DUFF 

University College London 
London, England 


INTRODUCTION 

In this paper, the terms “pattern recognition” and “image 
processing” will be considered only in relationship to optical 
or visual images, which will be expressed as two dimensional 
arrays of numbers, each number representing the image 
intensity in the corresponding picture element, or “pixel.” 
The image will be assumed to be adequately reproduced by 
means of an n by n array of square picture elements whose 
intensities are chosen from a range of L discrete values 
which are uniformly spaced over a selected brightness range. 
Obviously, both n and L must be large enough to give ac¬ 
ceptable spatial and grey-tone resolutions for the task to be 
performed on the image. 

Many of the operations carried out during pattern recog¬ 
nition or image processing can be described as “local;” this 
implies that the new value of a pixel in the processed image 
is a function of the values of a limited and compact subset 
of pixels in the original image. A particularly useful subset 
is the immediate neighbourhood of each pixel, i.e. the three 
by three array of adjacent pixels surrounding and including 
each pixel. The implication of this statement is that ex¬ 
tremely fast processing of images could be achieved by 
constructing an array of identical processors in one to one 
relationship with all the pixels of an image, each processor 
receiving inputs from its corresponding pixel and from the 
eight immediate neighbours of the pixel. The purpose of the 
CLIP (Cellular Logic Image Processor) programme of re¬ 
search has been to optimize a processor array structure for 
the range of operations commonly required in image proc¬ 
essing, having due regard to the cost of implementing the 
system as an array of large scale integrated circuits. At the 
same time, it was recognized that, ideally, the array should 
be capable of executing all possible functions provided an 
appropriate sequence of allowed instructions is obeyed. In 
other words, the array should perform as a general purpose 
computer optimized for a typical range of image processing 
operations. 

CLIP4 LOGIC 

The basic unit of the array is the cell, being a combination 
of logic units (i.e., the processor) and memory. Details of 
the CLIP4 cell are shown in Figure 1(a) in which A, B and 


C are single bits of buffer memory and D is a 32 by 1 bit 
RAM. The boolean processor provides two program se¬ 
lected independent functions, D and N, of the two binary 
inputs A and P, one of which (N) is used to form an output 
to all neighbouring cells. Inputs from neighbouring cells are 
individually gated and then OR’ed together to form the bi¬ 
nary function T which combines with B to produce P. A and 
B are each single bits of the pixels of two images in the 
array location corresponding with the cell concerned. Other 
gates are included which can be used to generate the exclu¬ 
sive OR (EXOR) of B and T at P and also the AND of B 
and T which is then OR’ed with the N output of the boolean 
processor to form the interconnection output N*. These 
additional gates are of use when the processor is performing 
arithmetic operations and are brought into use by special 
instructions which put l’s on the R and C control inputs. 

Before considering the details of the operations in the 
CLIP4 cell, it will be helpful to examine the organisation of 
the data storage in the n by n cell array (see Figure 1(b)). 
Each address in the D memories can be visualised as a plane 
of single bit stores, d. If the location of each cell is given by 
its (x,y) coordinate in the array, then a bit plane will be the 
set of bit stores defined by: 

Dj {djxy • x,y 1 , 2, . . . n\ 

A pixel will be formed as a column of single bit stores 
passing through g bit planes. Thus the pixel at location (x,y) 
is defined by: 

Pxv={d jxy : j=k, k+l, . . . k+ g- 1 } 

D k will contain all the least significant bits in the binary 
numbers representing the grey levels in each pixel; D k+g - y 
will contain the most significant bits. Note also that 

g=log 2 L 

implying g binary bit planes are needed to represent L grey 
levels. The ordering and addressing of the bit planes in the 
32 available addresses in the D memories is arbitrary and 
may be varied to suit the needs of the program. 

When numerical calculations are to be performed on data 
in which there is no longer an exact correspondence between 
the n 2 pixels and the n 2 processors, it is sometimes con¬ 
venient to represent data in the binary column mode, in 
which n binary numbers are stored in the n columns of a 
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single bit plane. A binary number will be represented by the 
column: 

{.dj X y. y 1,2, . . . , n} 

Normally, the precision provided by n bits will be unneces¬ 
sarily high for typical values of n. Methods have been de¬ 
vised for using the available storage more effectively. 

A binary image is composed of elements which are either 




Figure 1(b)—Bit plane and binary column data organisation 


black or white so that pixels have values 0 or 1 respectively. 
Usually, binary images are produced by thresholding a grey 
tone image. Binary images are stored in a single bit plane of 
the D memories, at any convenient address. 


USING THE CLIP4 CELL 

The five principal ways in which the CLIP4 cell can be 
configured are illustrated in the simplified logic diagrams of 
Figure 2(a)-(e). In each case, the symbols A, B, C and D 
refer to bit planes. 

Simple boolean operations (Figure 2(a)) 

Binary image F is loaded into A and binary image F 
loaded into B. The direction gates 1 to 8 and inputs R and 
C are disenabled. Any of the 16 possible boolean functions 
of the two images can be produced and loaded into a selected 
address of the D memories. If I t is in D 1 , and / 2 in D 2 , and 
if the result address is D 3 , the coding for the operation of 
taking the exclusive OR of F and / 2 is (in the CLIP4 lan¬ 
guage: CAP4): 

SET P @ A 
LDA 1 
LDB 2 
PST 3 

(Notes: @ implies EXOR; PST is a mnemonic for Process 
and Store) 
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BOOLEAN FUNCTION 
SELECTION INPUTS 



Figure 2(a)—Simplified logic for boolean operations 


Simple propagating operations (Figure 2(b)) 

A binary image is loaded into A and an interconnection 
function set up (for example: N=A). A selection of inter¬ 
connection gates are opened (for example: those in direc¬ 
tions such that outputs from neighbouring cells to the North, 
East, South and West are admitted, i.e. directions 2, 4, 6 
and 8) and a boolean function for the output D is chosen 
(say: P=A). The coding is: 

SET P + - A, [2 4 6 8] A 
LDA 1 
PST 3 


(Notes: the negative sign implies NOT; the B pattern does 
not affect the operation). 

Labelled propagating operations (Figure 2(c)) 

In this case propagation can be started from any cell in 
which B has the value 1, i.e., from any white pixel in / 2 . 
Using the same values as in the previous example, the cod¬ 
ing becomes 

SET P+-A, [2468B] A 
LDA 1 
LDB 2 
PST 3 

(Notes: the connection to the B store is enabled by inserting 
the symbol B at the end of the direction list on the SET 
instruction). 

Propagation can also be initiated at the edges of the array 
by adding an E at the end of the SET instruction. This has 
the effect of supplying a 1 to all interconnection inputs of 
array edge cells which cannot connect to missing neighbours 
lying outside the limits of the array. 

Bit plane arithmetic operations (Figure 2(d)) 

If A is a particular digit of one binary number, B the 
corresponding digit of another binary number, and C the 
carry digit resulting from the addition of the previous digits 
of less significance, then the expressions for the sum digit 
and new carry digit are: 

SUM=A.EXOR.B.EXOR.C 

CARRY=(B.EXOR.C).OR.B.AND.C 


D FUNCTION 

SELECTION 

INPUTS 



INTERCONNECTION 

INPUTS 

(Individually gated) 


INTERCONNECTION 
FUNCTION SELECTION 
INPUTS 


Figure 2(b)—Simplified logic for propagating operations 


INTERCONNECTION 

OUTPUT 
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D FUNCTION 

SELECTION 

INPUTS 



INTERCONNECTION 

OUTPUT 


Figure 2(c)—Labelled propagation logic 


When the inputs R and C are enabled, the CLIP4 cell will 
implement this arithmetic function for bit planes loaded into 
A and B, using the boolean processor to provide the oper¬ 
ations N=P.A and D=P.EXOR.A. If two grey tone images 
are loaded into D 7 to D 6 and D 7 to D l2 respectively, then 
the part of the coding which adds the second least significant 
digits is: 

SET P@A, [B] P.A, RC 
LDA 2 
LDB 8 
PST 14 

Giving a summed image in D 13 to D 19 . 

Binary column arithmetic operations (Figure 2(e)) 

Two planes of binary column numbers in D { and D 2 can be 
summed to form a new plane of binary column numbers in 
D 3 by writing the following short program: 

SET P@A, [6B] P.A,R 
LDA 1 
LDB 2 
PST 3 

(Notes: the carry function is automatic when R is enabled; 
no use is made of the C plane stores; direction gate 
6 is opened to allow carries to be propagated along 
the binary columns). 

Other arithmetic functions can be developed from the 
summing operation in the usual way. In principle, floating 


point arithmetic can also be performed, although full details 
of the algorithms involved have still to be worked out. 


OTHER HARDWARE FEATURES 

In the CLIP4 system now being constructed, a 96 by 96 
cell array is being assembled using custom-designed NMOS 
integrated circuits. Each circuit comprises eight cells ar¬ 
ranged in two rows of four. The A elements are serially 
connected to form 96 shift registers, each of length 96 and 
lying along the array rows. Single bit planes of data are 
loaded serially into a 9216 bit shift register external to the 
array; this shift register can then be reconfigured into 96 
short shift registers which then discharge their data into the 
A rows. The whole system interfaces to a television camera 
and monitor via two shift register memories, both of which 
can store a 96 by 96, six bit grey tone image. Analogue to 
digital conversion and digital to analogue conversion be¬ 
tween these memories and the television camera and mon¬ 
itor respectively proceeds at the full video rate. 

The array is operated by a controller which extracts the 
16 or 32 bit instruction words from the controller memory, 
decodes them and drives the array and peripherals. Included 
in the controller are 14 general purpose 16-bit registers. A 
fifteenth register has a special additional function: a bit plane 
may be shifted from the array through a tree of parallel 
adders, thus permitting a rapid count of the 1-bits set in the 
A stores. This count appears in register 15. Furthermore, 
another instruction can be used to output the contents of 
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Figure 2(d)—Bit plan arithmetic logic 


register 15 back into the array by means of an operation 
similar to the E instruction. 


CLIP4 INSTRUCTION TIMES 

CLIP4 is a Single Instruction, Multiple Data stream com¬ 
puter and performs image processing tasks at speeds which 
are quite unobtainable in conventional machines. The inte¬ 
grated circuit operates with a four phase, 400 ns clock cycle; 


load instructions require 8 cycles, PST instructions 10 cycles 
and SET instructions 12 cycles. A further 3 cycles are re¬ 
quired for each propagation step through a cell when the 
selected boolean function is such as to cause propagation. 
Branch and register instruction require 4 or 8 cycles (the 
longer time being for memory reference register instruc¬ 
tions). 

Interleaving of these instructions is used to reduce com¬ 
plete array operations to just under 9 /xs plus 1.2 /u,s/cell for 
propagation beyond the immediate neighbourhood of each 


SUM FUNCTION 
SELECTION INPUTS 



OUTPUT CARRY TO 
ADJACENT CELL 


Figure 2(e)—Binary column arithmetic logic 
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cell. In general, the complete operation will comprise the 
instructions SET, LDA, LDB and PST. A bit plane of data 
can be entered into the array in under 4 ms. However, this 
may increase up to 10 ms since the operation is synchronised 
with the television scan cycle. 


SYSTEM STATUS 

A small scale pilot model of the system (CLIP3) has been 
in operation since 1973. The array comprises 16 by 12 cells 
and can be used on images of the same size or, by scanning 
the array across a 96 by 96 pixel image, can be used to 
simulate the larger CLIP4 array. Additional hardware and 
software is incorporated to handle sector to sector propa¬ 
gation when CLIP3 is used in the scanned mode. Details of 
the CLIP3 system are provided in References 1-5, which 
also include bibliographies of earlier studies both at Univer¬ 
sity College and elsewhere. 

At the time of writing (late January 1978), the CLIP4 chip 
design is complete and prototype chips are expected to be 
delivered before the end of March 1978. All other parts of 


the hardware and software for the image processing system 
are complete, although some modifications and improve¬ 
ments are being carried out. Negotiations are in hand to 
arrange for CLIP4 systems to be manufactured and made 
commercially available. Preliminary estimates suggest that 
the first systems will be ready during mid to late 1979. The 
University College prototype is scheduled for operation in 
Autumn 1978. 
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Evolution of new hardware technology 


OVERVIEW 

Evolution of new hardware computer technology continues at a rapid pace. 
Implications of developments in semiconductor LSI technology for logic and 
memories, new achievements in magnetic bubbles and disk storage and progress 
in computer printer technology for a modern computer system are far reaching. 
The impact of these developments has resulted in (i) significant cost reduction 
for a given function; (ii) increased complexity resulting in enhanced capability 
and performance; (iii) improved reliability; and, (iv) reduction in physical size. 
These trends can be expected to continue during the next decade and even though 
these technological changes are evolutionary, their impact on the computer or- 
ganizaton and utilization is indeed revolutionary. Entire new areas of application 
will become available to the computer industry with the advent of inexpensive 
yet powerful and versatile hardware. The generic impact on the architecture and 
configuration of a computer will be as follows: 

• Every type of terminal, control and data entry equipment, etc., will have a 
rather large amount of intelligence at the point of use. 

• Dedicated small digital systems will be incorporated in computer subsystems, 
i.e., computer peripherals. 

• System design will be determined by repetitive use of very inexpensive 
hardware. 

• Archival mass storage will be available on line. 

• Today’s operating system features and other system software will be imple¬ 
mented in firmware. 

• Performance measurement monitoring, maintenance and error logging ac¬ 
companied with fail soft and fault tolerant design will be incorporated for 
increased hardware reliability and availability. 

The three key areas of technical progress responsible for the developments 
projected above are: (1) semiconductor technology for LSI logic, microprocessors 
and RAMs/ROMs; (2) magnetic technology for bubble memories, and disk stor¬ 
age; and, (3) printer technology for data/information output. Present status and 


1061 


1062 


National Computer Conference, 1978 


projected developments in these areas will be described in the sessions on hard¬ 
ware evolution. 

Semiconductor technology has been able to double the functional density on 
a single silicon chip every year since 1960. In the logic area, this progress has 
taken this technology from a single transistor to IC’s and then to microprocessors 
and microcomputers. In complexity the microprocessors have evolved through 
4 bit and 8 bit complexity to 16 bit complexity. By incorporating ever increasing 
RAM/ROM and control function powerful single chip microcomputers have be¬ 
come available for a price of about $10. Progress in the technology promises that 
by about the mid-eighties a 32 bit microcomputer with one million bits of memory 
could be made on a single silicon chip for a cost of about $20. With such 
inexpensive but powerful intelligence, no equipment would be permitted to stay 
dumb anymore. 

In the area of semiconductor memory, the first challenge to the dominance of 
magnetic core memories occured in 1970 when the first IK bit semiconductor 
memories became available. Since then the progress in this area has been rapid 
and now the technology stands on the threshold of the availability of 64K bit 
RAMs. This progression in complexity is likely to continue and reach a 1 million 
bits per chip by 1985. 

CCD and magnetic bubble technologies provide solid state alternatives to 
rotating electromechanical mass storage devices. During the past year, significant 
technical advances have taken place in both of these areas. Both of these tech¬ 
nologies have started to appear in the commercial products. CCD memories have 
faster access times than the bubble memories and will find a place in the memory 
hierarchy. Bubble memories, however, promise to reach the 1 million bit on a 
chip capability sooner than any semiconductor technology. The resulting low 
cost and high reliability when coupled with their inherent characteristics of non- 
volatility will permit this technology to fill the presently existing gap between 
semiconductor memories and moving magnetic mass storage. 

Even as the CCD and bubble memories become more cost effective and attempt 
to replace magnetic disk technology, the moving head disk technology continues 
to improve. Disk technology progress has included not only the readwrite heads 
but media, source coding and error correction and improved system bus inter¬ 
facing. These developments have improved the performance and reliability. Area 
density of storage has continuously increased and has been a key factor in 
achieving cost reductions. 

Printer technology developments is another area of major activity. Both the 
impact and non-impact printing technologies using plain paper have progressed 
significantly. Ink Jet and xerographic printers are now commercially available. 
Progress in these technologies is radically changing the quality of the computer 
output as well as the cost of computer printing. 

The technical developments in the areas discussed above will be described and 
projection of developments in the years to come will be made by experts who 
themselves have played a major role in making these developments possible. 

One more aspect of this rapid evolution in hardware technologies is that 
wherever there is a significant technological change, there are opportunities for 
entrepreneur organizations to bring these technologies to the marketplace. A 
panel of distinguished speakers will address themselves to the proposition of 
“Opportunities for New High Technology Companies’’ as a part of the series of 
sessions on Evolution of New Hardware Technology.’’ 
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by CHARLES BOETTCHER 

National Semiconductor 
Santa Clara, California 


Semiconductor memories have, since 1970, been a signifi¬ 
cant factor in data storage. This has been especially true in 
Random Access memories and is becoming true of the 
slower serial access memories. 

The manufacture of semiconductor memories takes place 
in two stages; the batch manufacture of the dice as wafers 
and the piece-by-piece assembly of the good dice into pack¬ 
ages. It is the batch manufacturing of the dice that has led 
to the continuing trend of lower cost per bit. (Figure 1) With 
each new generation of memories, four factors have con¬ 
tributed to increased density of good bits per batch and 
therefore lower cost. These four factors are; decrease in 
storage cell complexity, decrease in feature size, increase in 
wafer size (thus batch size), and decrease in defect density. 

Decrease in storage cell complexity has been the most 
significant factor in increasing batch density. (Figure 2) An 
appropriate comparison of cell complexity can be made by 
expressing cell area in units of / 2 , where /is the minimum 
feature size allowed by the pattern-definition technology. 
Cell sizes in the last eight years have decreased from 200/ 2 
for the first static flip-flops to a present range of 16/ 2 to 
20 f 2 . It is theoretically possible to reach a cell size of 4/ 2 
where there are two features in both the X and Y directions, 
one to store the information and the other to isolate it from 
adjacent cells. If means can be found to isolate adjacent 
cells in less than a feature size, then a cell size of / 2 is 
possible. 

Added to the decrease in cell size is an inherent increase 
in layout efficiency. When memory size increases by a factor 
of four, only twice as many decoders, sense amplifiers, etc., 
are required; only two more address buffer circuits and the 
same number or less bonding pads are needed. 

A combination of design and process innovation has al¬ 
lowed this decrease in storage cell complexity, resulting in 
a 13.5 times increase in batch density. 

The second most significant of the four factors has been 
increased wafer size. As a result of a significant amount of 
development work on the equipment for manufacturing raw 
wafers and processing wafers, as well as developing proc¬ 
essing techniques for handling larger wafers, the size of 
wafers has increased from two inches to four inches in 
diameter. This has resulted in a four times increase in wafer 
area and therefore a four times increase in batch density. 

The remaining two factors, feature size and defect density 


have contributed the least to increasing the number of good 
bits per batch. Smaller minimum feature sizes have been 
possible as a result of improvements in mask making, mask 
aligning, and photoresist technology. Full utilization of im¬ 
provements has been hampered by several factors. As fea¬ 
ture sizes approach the film thicknesses and step heights 
present, dimensional control and feature integrity becomes 
increasingly difficult to maintain. In addition, as the feature 
size is reduced, the size of particle that will cause a defect 
in the masking is also reduced. Since smaller particles have 
a much higher probability of passing through the various 
forms of filtering used in semiconductor manufacturing fa¬ 
cilities, the effective density of defects increases exponen¬ 
tially with decreasing feature size. (Figure 3) These factors 
have limited batch density improvements due to feature size 
reduction to a 2.2 times increase. 

Overall, these four factors have resulted in a 120 times 
increase in batch density. With each new generation of mem¬ 
ories, when a four times larger memory could be fabricated 
and packaged for the same cost as four smaller memory in 
four packages, the new generation was introduced. The 
packaging density increase of course offered an overall sav¬ 
ings in system costs and users started switching to the new 
generation. As volume has increased and yields improved, 
costs have dropped and the cycle has repeated. 

To what extent can we extrapolate and thereby predict 
the future? 

With respect to decreasing cell complexity, there is little 
real room for progress. Present design and process technol¬ 
ogy is producing cells in the 16/ 2 to 20/ 2 size range. If we 
attain a 4p cell size, we will only realize a factor of four to 
five times density increase. In addition, in order to attain a 
4 f 2 cell size, significantly new device types will have to be 
developed or adjacent storage cells will have to be built on 
separate layers. Increasing the number of layers affects yield 
in the same manner as a larger area. Therefore, achieving 
a 4 p cell, with a corresponding increase in number of mask 
layers, may give no actual improvement in good bits per 
batch. Examples of new device types are Ovonic materials 
sandwiched between layers at the memory crosspoints or 
CCD devices which require changes in system architecture 
to cope with large percentages of their data base in serial 
access storage. In such cases the technology development 
required is significant. 
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NOTES: 

(1) Product is introduced, very little or no competition. 

(2) Product is multiple sourced but prices still hold up as 
production is limited. 

(3) Production expands dramatically as yields improve, com¬ 
petition is fierce, prices tumble, marginal suppliers are 
beginning to drop out. 

( 4 ) Significant yield improvements are no longer available, new 
product is beginning to compete for the same market, 
competition is still intense, suppliers still drop out. 

(5) Major market share is taken by new generation of product, 
market shrinks but prices hold up as competition is minimal. 

Data courtesy of T. Klein, National Semiconductor. 

Figure 1—Typical decrease of memory products’ prices/costs during 
product life 


With respect to wafer size increase, the trend will con¬ 
tinue, wafer area will double every four years. Each time 
manufacturers change wafer sizes, virtually all wafer proc¬ 
essing equipment has to be replaced at a tremendous cost. 
Planning ahead and installing equipment that has a longer 
life is not really possible because of the ability with which 
equipment can be developed that will handle larger wafers 
with increasingly stringent control requirements. 

Feature size reduction is the area where the industry is 
predicting the most improvement in batch density. 

Projection aligners are already in use or being installed in 


most manufacturing lines. They have allowed the masks to 
become permanent tooling rather than production materials 
that must be discarded with every second or third run, and 
therefore, made it practical to invest in defect free masks 
with very tight tolerances. Projection aligners will allow 
feature sizes approaching one to two microns, a five times 
improvement over present practice. The lower defect dens¬ 
ities possible with projection aligners will help offset the 
exponentially increasing effective defect density inherent 
with smaller feature size, but significant and costly improve¬ 
ments will also be needed to reduce other defect sources. 

Electron Beam exposure systems are being developed, 
evaluated and purchased. Proponents of E.B. systems are 
claiming that they are about to revolutionize the industry. 
Proper analysis indicates that they will be a useful tool for 
mask making purposes, but their costs and throughput are 
such that feature size improvements they make possible will 
be offset by increased processing costs. 

The industry is probably five to eight years away from the 



YEAR 

NOTES: 

11) Theoretical limit, current wafer and feature sizes. 

(2) Theoretical limit, assuming continued improvements in 
wafer size increase and feature size reduction. 

(3) Theoretical limit, assuming constant rate of wafer size 
increase and an increasing rate of feature size reduction. 

(4) Increase due to design and process innovations. 

(5) Increase due to feature size improvement. 

(6) Increase due to wafer size. 

Data courtesy of T. Klein, National Semiconductor. 

Figure 2—Past and projected contributions to batch density improvements 
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time when the cost and throughput of E.B. systems will 
allow practical use in direct wafer manufacture. 

Feature size reduction, alone, does not determine the size 
of features that will be used, only the lower limit available. 
The devices from which the circuits are constructed must 
be scaled within other physical constraints. 

Theoretically, MOS transistors, for example, can be 
scaled to quarter micron feature size, but some device char¬ 
acteristics don’t scale well. As devices are scaled, circuit 
operating voltages and power densities must scale propor¬ 
tionately. 

Such scaling requires re-engineering of device structures, 
process control, and film thicknesses. Parasitic parameters, 
such as leakage, noise, sub-threshold current, and sheet 
resistance do not scale as other circuit parameters and will 
require innovative design and/or materials research to deal 
with them. 

Considering these factors then, it is theoretically possible 
for the batch density to increase by another 100 times in the 
next eight years. The task will require very large investments 
in capital and talent. The number of manufacturers able to 
make these investments will decrease. Moreover the market 
for memory products must continue to expand at a rate that 
will allow amortization of those investments. 

It is appropriate at this point to apply this information to 
predictions of the coming generations of semiconductor 
memories. 

In dynamic RAMs, MOS will dominate. The 16K will be 
the main production part through mid-1980. By mid-1980 
significant quantities of 64K RAMs will be available. The 
64K Dynamic RAMs will use scaled MOS devices (4 micron 
feature size), reduced power supply voltages (probably 5 
Volts), 10 f 2 to 12/ 2 cell sizes, and be manufactured on 4 
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Figure 3—Effective density of catastrophic defects vs. feature size 
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Figure 4 —Memory component cost vs. performance 


inch wafers. Some of these same techniques may be applied 
to the 16K RAMs but the cost of packaging four 16K RAMs 
will be offset by the lower cost of 64K RAMs, both for the 
manufacturer and for the user. 

Storage cells for CCD memories are inherently easier to 
simplify and already 64K CCD’s with 5 Vsf 2 cell sizes are 
close to reality. The disadvantages of serial memory storage 
will be offset by the lower cost per bit associated with 
CCD’s. Memory system and CPU manufacturers will accept 
the 5 to 10 percent throughput penalties in exchange for Vs 
to V\ the cost per bit CCD’s will offer. 

MOS static RAMs will use scaled MOS to achieve data 
rates presently served by bipolar technology. They will be 
used predominantly to serve the IBM Add-on business until 
the problems of interfacing Dynamic RAMs to the new gen¬ 
erations of machines are solved. Currently, 4K static RAMs 
are becoming readily available. By mid-1980 16K static 
RAMs will be readily available. Present CPU’s are imple¬ 
mented with ECL logic, and the translation from ECL to 
TTL levels to interface the RAMs adds appreciably to the 
access time. Future generations of MOS static RAMs will 
appear with ECL interfaces to eliminate the external level 
shifting and potentially reduce access times. 

Bipolar static RAMs will continue to be the faster and 
smaller density memory devices. The bulk of the products 
will be ECL due to interfacing and access time requirements. 
Products will range from sub-five nanosecond 128-bit mem¬ 
ories to sub-twenty-five nanosecond 4K bit memories. Cell 
sizes will be relatively large (120/ 2 ) because of scaling dif¬ 
ficulties and complex cell designs. Wafer size will be four 
inches. Device development will be concentrated towards 
reduction of parasitics (i.e., passive isolation) and increased 
gain band-width product of the transistors. 
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In summary, Bipolar static RAMs will move to higher 
speeds, MOS static RAMs will fill the speed range presently 
served by Bipolar RAMs (at lower cost and higher density), 
MOS Dynamic RAMs will be denser and lower in cost per 
bit, and CCD’s will extend the range of semiconductor mem¬ 
ories in the direction of longer access time and lower cost 


per bit. (Figure 4) The investments in terms of talent and 
capital will be large for the manufacturers able to stay in the 
competition. 

Grateful acknowledgment goes to Thomas Klein for his 
analysis of Batch Density relationships. 


© 1978 National Semiconductor Corporation 



Bubbles and CCD memories—Solid state mass storage 


by J. EGIL JULIUSSEN 

Texas Instruments Incorporated 
Dallas, Texas 


INTRODUCTION 

In the last year significant technical advances have taken 
place in the development of magnetic bubble memories 
(MBM) and charge coupled devices (CCD). The first bubble 
memory chip has been introduced with bigger and better 
chips under development. Although 16K bit CCD chips have 
been available for two years, it is the recently announced 
64K bit chips which will make an impact on computer sys¬ 
tem designs. The first commercial products based on MBM 
and CCD chips are now starting to appear. 

With this progress it is worthwhile to look at the status of 
CCD and bubble memories, their applications and future 
potential. 


MBM AND CCD STATUS 

Both CCD and bubble memories are integrated circuit 
analogs to rotating electromechanical memories such as 
disks and tapes. CCDs store information in rows of capac¬ 
itors. Bubbles and electromechanical memories store infor¬ 
mation as rows of magnets. The access methods are differ¬ 
ent. Electromechanical memories move the media while the 
magnetized regions remain stationary. In CCD and bubble 
memories the opposite takes place: the media remains sta¬ 
tionary while the information, charge or magnets, move 
within the media. This avoids all mechanical motion and its 
associated drawbacks. The result is two solid state mass 
storage technologies—or electronic disks. For further infor¬ 
mation on the technical aspects of MBM and CCDs see 
References 1-10. 

The features of MBM and CCDs are shown in Table I. 
Both technologies are serial access memories and are or¬ 
ganized as shift registers. Bubble memories are nonvolatile 
as the information is retained without power. The electrical 
analogy of the magnetic bubble, the CCD, does not keep its 
charge or information as the power is turned off. CCDs are 
volatile and must be periodically refreshed to retain the 
information. 

Both technologies have the modularity advantages arising 
from their chip packaging. The resulting low entry price is 


very important for small systems. As chips they can also be 
packaged directly with the CPU on PC boards. 

The characteristics of today’s major CCD and bubble 
memory components are shown in Table II. The 92K bit 
MBM chip has an average access time of 4 milliseconds and 
has a transfer rate of 50K bits per second. The dual-in-line 
package is larger than most ICs. The MBM package includes 
two permanent magnets giving non-volatility and two coil 
windings. The two coils produce a rotating magnetic field 
which moves the bubbles around the shift registers. 

MBM support chips are already available. A minimum 
bubble memory system which is interfaced to a micropro¬ 
cessor can be made with 7 chips. The coil drivers produce 
the rotating magnetic fields. The sense amplifier detects the 
bubble information and is very similar to a core memory 
sense amplifier. The function timer and driver produce the 
necessary timing signals to control the operation of bubble 
memory chips. The controller chip is a LSI device which 
controls the memory system operations, buffers data and 
handles interfacing with a microprocessor. 

The 64K bit CCD chips have a superior access time over 
bubble memories. The TI 9 and Fairchild devices have a 410 
microsecond average access time and can transfer data at 1 
to 5 Mbits per second rate. The Intel device has more shift 
registers with shorter length that gives better access time, 
but at a lower transfer rate. During operation the CCD chips 
use less power than bubble memory chips. The power ad¬ 
vantage is reversed to MBMs in a standby mode. 

None of the CCD products have any support chips. It 
should be noted that CCDs do not need an external sense 
amplifier. This function and the refresh function have been 
integrated onto the chip (one per shift register). 

An assessment of MBM and CCD merits are shown in 
Table III. The main advantages of bubble memories are the 
nonvolatility and the large number of bits per chip. In effect 
MBM is the only solid state technology which is nonvolatile. 
The major advantages of CCDs are the low access time and 
the high transfer rate. 

Bubble memories have an advantage over CCD from 
fewer manufacturing steps and fewer masking steps. This is 
also reflected in a potential higher MBM packaging density. 
However, the experience from MOS manufacturing is very 
applicable to CCD production and probably outweighs the 
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TABLE I.—MBM and CCD Features 

MBM 


• SERIAL ACCESS MEMORY 

I SHIFT REGISTER ORGANIZATION 

• BLOCK ADDRESSABLE 

• NONVOLATILE STORAGE 

• READ-MODIFY-WRITE 

• MODULAR STORAGE CAPACITIES 

0 LOW ENTRY PRICE 

0 BIT PRICE INDEPENDENT 

OF STORAGE SIZE 
0 FEW MANUFACTURING STEPS 

0 MANUFACTURE SIMILAR TO IC PRODUCTION 

0 STOP/START OPERATION 

0 LOWERS EFFECTIVE ACCESS TIME 

0 LOWERS POWER DISSIPATION 

0 VARIABLE TRANSFER RATE 

0 MINIMAL DATA BUFFERING 


CCD 

0 SERIAL ACCESS MEMORY 

0 SHIFT REGISTER ORGANIZATION 

0 BLOCK ADDRESSABLE 

0 VOLATILE STORAGE 

0 READ-MODIFY-WRITE 

0 MODULAR STORAGE CAPACITIES 

0 LOW ENTRY PRICE 

0 BIT PRICE INDEPENDENT 

OF STORAGE SIZE 
0 TTL COMPATIBLE I/O 

0 MOS COMPATIBLE MANUFACTURING 

0 STORAGE REFRESH REQUIRED 


MBM advantage today. As bubble memory producers gain 
experience, it will be interesting to see if the potential MBM 
manufacturing advantages can be realized. 

In packaging CCDs have an advantage due to the added 
complexity of the coils and magnets for a MBM chip. 

In interfacing complexity MBM currently has an advan¬ 
tage from the availability of support chips. The low transfer 
rate and these support chips are very advantageous for mi¬ 
croprocessor based systems. The high performance require¬ 
ments of mainframe and supermini computers give CCDs a 
distinct interfacing advantage for these applications. 

The bottom line is price. In the next few years, CCD will 
probably have an advantage due to MOS manufacturing 
experience and CCD availability from three or more manu¬ 
facturers. With increased bubble memory manufacturing ex¬ 
perience, MBM should close this gap and have an excellent 
chance of gaining an advantage over CCDs. 

Table IV shows the announced MBM and CCD products 
the author is aware of. There are undoubtedly more products 
under development. 

The three first MBM products use TI’s 92K bit bubble 
memory chip. The TI 763/765 portable bubble memory ter¬ 


minals are part of TI’s Silent 700®* series of terminals. Both 
terminals have a minimum of 20K bytes of bubble memory 
with optional expansion to 80K bytes. 11 

Q1 corporation has a microcomputer system with 80K 
bytes of bubble memory and also has a plasma display. Data 
Systems MBM system is a floppy disk replacement which 
is PDP-8 and PDP-11 plug compatible. Storage capacities of 
86-519 Kbytes are available. 

AT&T’s 13A announcement system uses 68K bit bubble 
memory chips developed by Bell Laboratories and manu¬ 
factured by Western Electric. The system stores pre-re¬ 
corded messages which are repeatedly played back as tele¬ 
phone system messages. The system is being field tested by 
AT&T. 12 

The three CCD systems use Intel’s 16K bit CCD chips. 
Intel’s CCD board is an OEM product that can be used to 
develop end user systems. Technical Analysis Corporation 
has developed a fixed head disk replacement for their Nova 
based small business computer. 13 Alpha Data also has a 


* ®Registered trademark. 
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TABLE II.—MBM and CCD Chip Characteristics 



TEXAS INSTRUMENTS 

FAIRCHILD 

F 464 

CCD CHIP 

INTEL 

2464 

CCD CHIP 

TIB 0103 

MBM CHIP 

TMS 3064 

CCD CHIP 

STORAGE CAPACITY 

92 KBIT 

64 KBIT 

64 KBIT 

64 KBIT 

AVG ACCESS TIME 

A MSEC 

410 SEC 

410 SEC 

130 SEC 

MAX TRANSFER RATE 

50 KB/SEC 

5 MB/SEC 

5 MB/SEC 

2.5 MB/SEC 

POWER 

0.7 W 

0.26 W 

0.34 W 

0.33 W 

STANDBY POWER 

NONE 

25 MW 

66 MW 

41 MW 

OPERATING TEMP 

0 TO 70° C 

0 TO 70° C 

0 TO 55° C 

-10 TO 85° C 

PACKAGING 

14-PIN DIP 

16-PIN DIP 

16-PIN DIP 

18-PIN DIP 


Q" x 1.1") 




PRODUCTION STATUS 

PRODUCTION 

PRODUCTION 

PRODUCTION 

PRODUCT ANNOUNCED 

SUPPORT CHIPS 

CONTROLLER CHIP 




AVAILABLE 

COIL DRIVER 



NONE 


SENSE AMPLIFIER 





FUNCTION DRIVER 





FUNCTION TIMER 





fixed head disk replacement which is compatible with their 
other fixed head disk products. 

There are many products in the development stage which 
use the newer 64K bit CCDs, but none are announced yet. 

To summarize the status of CCD and bubble memories a 
list of the companies with known R&D efforts are shown in 
Table V. As could be expected the major semiconductor 
manufacturers appear on the CCD list. Until recently only 
TI of the semiconductors producers had bubble memory 
effort. In the last year both Intel and National have started 
bubble memory efforts. 

MEMORY SYSTEM TRENDS 

It has been the goal for computer systems to have a single 
memory technology that has both the lowest cost and the 
highest performance. This goal has not been realized and 
the memory hierarchy from fast semiconductor RAMs to 
low cost disks and tapes must be used. The well-known 
memory hierarchy gap in access time and cost has provided 


a market pull for new memory technologies. In the early 
seventies this was the major driving force to develop CCD 
and bubble memories. 

Additional market pull forces are now equally important. 
The proliferation of microprocessor applications are de¬ 
manding small, non-volatile mass memory systems. Severe 
environments and the high maintenance costs need im¬ 
proved reliability. The popularity of virtual memory operat¬ 
ing systems need some real and high performance mass 
memory. 

The trend towards using fixed heads in combination with 
moving head disks is one solution that gives higher perform¬ 
ance mass storage. However, there is still a large percentage 
of computer systems that are disk-bound. 

MBM and CCDs are not removable storage technologies 
and this is a drawback vis-a-vis disks and tapes. There is a 
trend towards using non-removable disks. This lowers the 
disk drive cost and also improves reliability. The trend is 
also advantageous for both MBM and CCDs as both tech¬ 
nologies can replace the function of non-removable disks. 
At the same time, the use of distributed processing systems 
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TABLE III.—MBM Versus CCD 



MBM 

CCD 

MAJOR ADVANTAGES 

NONVOLATILE 

ACCESS TIME 


BITS/CHIP 

TRANSFER RATE 

MANUFACTURING 

FEW STEPS 

FEW MASK LEVELS 

EXPERIENCE FROM MOS 
MANUFACTURING 

PACKING DENSITY 

ADVANTAGE DUE TO MASK LEVELS 


PACKAGE 

COILS 8 MAGNETS ADD COMPLEXITY 

SIMPLE DIP 

INTERFACING 

ADVANTAGE FOR MICROPROCESSORS 

ADVANTAGE FOR MAINFRAMES 

PRICE 

ADVANTAGE BASED ON 

PACKING DENSITY 

ADVANTAGE FROM MOS 
MANUFACTURING EXPERIENCE 


also lowers the need for removability. The data communi¬ 
cation links in many cases are a substitute for removability. 

To indicate how MBM and CCDs compare with other 
storage technologies, the system prices for end users are 
shown as a function of storage capacities in Figure 1. For 


PURCHASE 
PRICE ($) 



STORAGE CAPACITY (MEGABITS) 

Figure 1—1978 memory system prices including interfacing 


small storage capacities MBM and CCDs have cost and 
physical size advantages. As soon as a few megabits are 
needed, the disks have a price advantage, but MBM and 
CCDs retain their performance edge. It is expected that 
MBM and CCDs, in the future, will be price competitive at 
higher and higher storage capacities. 


MBM AND CCD APPLICATIONS 

To gain insight in the use of CCD and bubble memories 
Table VI shows the various storage peripheral product cat¬ 
egories for mainframe, mini and microcomputers. MBM and 
CCDs are not fast enough to be used as main memory, and 
are too costly to be used for removable mass storage. As 
fast auxiliary memory (FAM) both MBM and CCDs have 
excellent credentials. This is the product category which 
fills the memory hierarchy gap. In the non-removable mass 
storage category, MBM and CCDs are very viable, but will 
see strong competition from disks. 

Due to their packaging flexibility, MBM and CCDs add 
an extra dimension to mass storage applications. They can 
be packaged directly with the processor and become an 
integral mass storage device. No extra box is needed as is 
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MBM 


TABLE IV.—Announced MBM and CCD Products 

_CCD 


• TEXAS INSTRUMENTS I 

• TI 763/765 PORTABLE BUBBLE 
MEMORY TERMINAL 

• 20 TO 80 KBYTES MBM 

I 

I QI CORPORATION 

• 01/LITE MICROCOMPUTER 

I 80 KBYTES MBM 

I DATA SYSTEMS | 

• DSD 6N0 FLOPPY DISK REPLACEMENT 

• 86 TO 519 KBYTES MBM 

I PDP-11 8 PDP-8 PLUG COMPATIBLE 

I A T*T 

I 13A ANNOUNCEMENT SYSTEM FOR 

RECORDED SPEECH 

• 68 KBYTES OF MBM FOR 20 SECONDS 
OF SPEECH DATA 


INTEL 

• IN-65 CCD BOARDS 

• 128K TO 1 MBYTES CCD 

TECHNICAL ANALYSIS CORP. 

» FIXED HEAD DISK REPLACEMENT 

• 256K TO 2 MBYTES CCD 

I DG NOVA PLUG COMPATIBLE 

ALPHA DATA 

I CCDISC - FIXED HEAD DISK REPLACEMENT 

• 128K TO 1 MBYTES CCD 


usually the case with disks and tapes. Instead the MBM or 
CCD can be put on the same or different PC board as the 
processor. The TI 763/765 portable bubble memory terminal 
is an excellent example of this advantage. 

The most likely CCD and bubble memory applications are 
shown in Table VII. The main bubble memory applications 
are in small systems. The bubble memory characteristics fit 
microperipherals and integral mass storage applications very 
well. In the miniperipheral category, MBM add-in or add¬ 
on electronic disks are viable opportunities. Add-in refers 
to MBM packaged with the CPU, while add-on means a 
separate box. In essence MBM is a main candidate for 
distributed mass storage to go along with distributed proc¬ 
essing. 

The main CCD applications are in closing the memory 
hierarchy gap. These opportunities are mostly in the large 
computer market place. In other words, CCD will be the 
technology which makes virtual memory systems become 
real. 

With MBM and CCDs having similar features, will they 
compete head-on for the same applications? To answer this 


question, Table VIII shows the probable MBM and CCD 
usage. 

The high performance characteristics will favor CCDs in 
mainframe computer applications. CCDs for fixed head disk 
replacement have already seen usage. Virtual memory and 
moving head disk cache memory based on CCDs are prob¬ 
ably in development. 

The nonvolatility and support circuit availability of bubble 
memories will favor MBMs as integral mass storage for 
microprocessor based systems. The announced MBM prod¬ 
ucts fall in this category, and more are in the development 
stage. 

The effect is that MBM and CCDs will have little direct 
application competition. 

In summary, the usage of MBM and CCDs will evolve as 
follows: In 1977 many system designers gained familiarity 
with the two technologies. In 1978 CCD and bubble mem¬ 
ories are designed into many products. In 1979 numerous 
products with MBM and CCDs will be in production. In the 
1980’s both MBM and CCDs will have become important 
storage technologies. 




TABLE V.—Companies with MBM or CCD Effort 


MBM 


CCD 


• US COMPANIES 

I TEXAS INSTRUMENTS 

• BELL LABORATORIES 

I IBM 

• ROCKWELL INTERNATIONAL 

• HEWLETT-PACKARD 

I UN IVAC 

• INTEL 

. • NATIONAL 

• FOREIGN COMPANIES 

• FUJITSU 

• HITACHI 

• NIPPON ELECTRIC 

• PHILIPS 

I PLESSEY 


I US COMPANIES 

• TEXAS INSTRUMENTS 

• BELL LABORATORIES 

I IBM 

I FAIRCHILD 

I INTEL 

• NATIONAL 

• MOSTEK 

• MOTOROLA 

• HUGHES 

• RCA 


FOREIGN COMPANIES 

BELL NORTHERN • 
PHILIPS • 
PLESSEY • 


SIEMENS 
TOSHIBA 
NIPPON ELEC1 
MITSUBISHI 


TABLE VI.—Storage Peripherals 



MAIN MEMORY 

FAST AUXILIARY 

MEMORY 

NONREMOVABLE 

MASS STORAGE 

REMOVABLE MASS STORAGE 

ON-LINE 

OFF-LINE 

PERIPHERALS 

CORE 

MOS RAM 

BIPOLAR RAM 

FIXED HEAD DISK 

EBAM 

CCD 

MBM 

MULTIPLATTER 

DISK 

EBAM 

CCD 

MBM 

MULTIPLATTER 

DISK 

AUTOMATED TAPE 

SYSTEM 

MAGNETIC 

TAPE 

MINI¬ 

PERIPHERALS 

CORE 

MOS RAM 

FIXED HEAD DISK 

CCD 

MBM 

SINGLE PLAT¬ 
TER DISK 

MBM 

CCD 

SINGLE PLAT¬ 
TER DISK 

FLOPPY DISK 

MAGNETIC TAPE 

TAPE CARTRIDGE 

FLOPPY DISK 

MICRO- 

PERIPHERALS 

MOS RAM 

ROM 

PROM 

EPROM 

MBM 

CCD 

MBM 

CCD 

MINIFLOPPY 

CASSETTE 

MINIFLOPPY 

MINICASSETTE 
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TABLE VII.—MBM and CCD Applications 


MBM 

I MICROPERIPHERALS ~ _ ~ 

• MICROCOMPUTER MASS STORAGE 

• FLOPPY DISK REPLACEMENT 

I INTEGRAL MASS STORAGE 

• PROGRAMMABLE TERMINALS 

I PORTABLE TERMINALS 

• WORD PROCESSING TERMINALS 

• BANKING TERMINALS 

I POINT-OF-SALE TERMINALS 

• DATA COLLECTION TERMINALS 

• PROGRAMMABLE CALCULATORS 

• MEASUREMENT 8 TEST EQUIPMENT 

• MINIPERIPHERALS 

I ELECTRONIC DISK 

I FIXED HEAD DISK REPLACEMENT 

• OPERATING SYSTEM STORAGE 

• MAINFRAME PERIPHERALS 

• FAST AUXILIARY MEMORY 

I DISK CACHE 

WHAT’S AHEAD IN CCD AND MBMs 

CCD and bubble memories will follow the rapid increase 
in storage capacities and with the resulting decrease in cost 
per bit which is the characteristic of semiconductor tech¬ 
nologies. Every two to three years the storage capacity of 
semiconductor chips quadruple and similar price per bit 
reductions take place. Both MBM and CCD chips will im¬ 
prove in this manner. In the beginning new storage tech¬ 
nologies improve even faster and CCD and MBMs are likely 
to do so. 

An overview of solid state memory trends is shown in 
Figure 2. 14 It shows that a 256K bit bubble chip can be 
expected in a year with a 256K bit CCD chip following a 
year later. By 1980 a 1 Mbit bubble memory chip is fore¬ 
casted with the 1 Mbit CCD chip following a year later. 

In terms of performance, CCD chips will still have an 
edge in access time and transfer rate over bubble memories. 
Both technologies will have higher transfer rates than today. 
The access time will be a trade-off parameter. To make 
larger chips will require longer shift registers. This would 
result in longer access time, unless a faster shift rate is used. 


_CCD_ 

• MAINFRAME PERIPHERALS 

• VIRTUAL MEMORY 

• DISK CACHE 

• FIXED HEAD DISK REPLACEMENT 

I MINIPERIPHERALS 

I ELECTRONIC DISK 

• OPERATING SYSTEM STORAGE 

• VIRTUAL MEMORY 

I FIXED HEAD DISK REPLACEMENT 

I MICROPERIPHERALS 

• MICROCOMPUTER MASS STORAGE 

• FLOPPY DISK REPLACEMENT 

• BUFFER STORAGE 

• CRT REFRESH 

• HIGH SPEED BUFFERING 


It appears that for both MBM and CCD’s the increased shift 
rate will just keep up with the longer shift registers. The 
result is that the access time of MBM and CCDs will be 
about the same as they are now. 

But how will MBM and CCDs stack up against disks? The 
moving head disk technology has had impressive advances 
in the last 15 years, and will probably keep on improving 


BIPOLAR RAM 

® 
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Figure 2—Solid-state memory technology trends 
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TABLE VIII.—MBM and CCD Usage 



MBM 

CCD 

COMMENT 

MAINFRAME COMPUTERS 

FHD REPLACEMENT IS A 
POTENTIAL APPLICATION 

FHD REPLACEMENT IS 

AN IMMEDIATE 
APPLICATION 

CCD'S MAJOR 
NEAR-TERM APPLICATION 


ELECTRONIC DISKS AWAIT 
MBM PRICE REDUCTION 

VIRTUAL MEMORY AND 

MHD CACHE WILL 
EMERGE 


MINICOMPUTERS 

SMALL ADD-IN ELECTRONIC 
DISK WILL EMERGE 

SMALL ADD-IN FAMS 
WILL SOON BE 
AVAILABLE 

APPLICATION FOR 

BOTH MBM AND CCD 


ELECTRONIC DISKS 
COMPETITIVE WITH 
CARTRIDGE DISKS MAY 
APPEAR BEFORE 1980 



MICROSYSTEMS 

INTEGRAL MASS STORAGE 

FOR MPU-BASED 
SYSTEMS IS AN IMMEDIATE 
APPLICATION 

INTEGRAL MASS STORAGE 

FOR SOME 
APPLICATIONS 

MBM'S MAJOR 

NEAR-TERM 

APPLICATION 


for another five to ten years. The disk advances have been 
realized by packing an increasing number of bits in the same 
area. The result is that large multi-platter disks that stored 
2M bytes in the early 1960’s now store 300M bytes, and cost 
about the same. Simultaneously lower priced units have 
been developed. The cartridge or single platter disk ap¬ 
peared in the late 1960’s and now stores about 100M bits. 
The floppy disk was introduced in the early 1970’s and just 
2 years ago the minifloppy disk was introduced. 

Impressive as these improvements are, the CCD and bub¬ 
ble memories will become increasingly competitive with 
disks. The disk technology is vulnerable because its bit price 
is very dependent on the storage capacity. The bit price of 
a minifloppy is about 50 times higher than the bit price of 
large disks. By 1980-81 a single MBM or CCD chip will 
store more than today’s minifloppy disk. The single unit 
OEM price of the minifloppy is about $350 today and may 
decrease another $100 in the next couple of years. This will 
be no match for the learning curve pricing of MBM and 
CCDs. An analogy of what is likely to happen is the micro¬ 
processor price which has decreased from several hundred 
dollars to less than ten in just three years. 

In summary, it appears that CCD and bubble memory 


systems will be price competitive with floppy and cartridge 
disks by the early 1980’s. 


SUMMARY 

For a new storage technology to succeed it must meet sev¬ 
eral requirements. The technology must have advantageous 
features and characteristics including competitive price/per¬ 
formance. Long term technology improvement potentials 
must exist. Significant R&D investment in the technology 
by many companies is crucial. There must, of course, be 
available markets. Equally important is an initial market 
entry niche. 

Both CCD and bubble memories meet these requirements 
and should therefore succeed in becoming major storage 
technologies in the 1980’s. 
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Programming and operating systems 


Four very different aspects of the areas of programming languages and oper¬ 
ating systems are represented in these sessions. 

A unique session will be a report on the recent History of Programming 
Languages Conference. No single topic seems to generate as much varied and 
heated debate as the merits of a programming language. Since future generations 
learn of past events through historical documents, special care had to be taken 
in organizing a History of Programming Languages Conference to insure a bal¬ 
anced and fair treatment of the various topics. An attempt will be made to 
summarize the technical lessons learned. 

The 1970’s have seen increasing interest in secure computing systems. Both 
theoretical results and practical experience with our systems seem to be some¬ 
what discouraging. Nonetheless, our intrepid designers keep trying. A report will 
be made which focuses on how close we are to achieving such systems. 

There has been a flurry of recent work on cryptography which suggests that 
improved and economical ciphers are available. Different aspects of such work 
will be discussed. One is the NBS standard while another concerns the use of 
data dependent keys. 

There will be a session of the status of COBOL which will discuss, among 
other things, the anticipated 1980 standard and also data base interfaces. 





Issues in kernel design* 


by GERALD J. POPEK and CHARLES S. KLINE 
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INTRODUCTION 

As operating systems became larger and more complex, 
interest increased in segmenting that software in a rational 
fashion. With respect to operating systems, a number of 
efforts were made to develop a low level base that would 
provide basic system functions in a highly reliable way. On 
top of this skeletal software base, extensive operating sys¬ 
tem functions and user supports would be built. Early efforts 
in this direction, although significantly different in scale and 
goals, include IBM’s virtual machine system CP-67, 1 Brinch- 
Hansen’s RC-4000 nucleus, 2 and CAL-TSS. 3 

However, the greatest impetus to this activity grew re¬ 
cently out of the desire for secure operating systems—those 
that could assure that the access to data stored within was 
controlled and protected in an uncircumventible way. Ef¬ 
forts have been made by a number of groups to design and 
develop operating system bases that, in addition to providing 
necessary primitive functions, were wholly responsible for 
the security of the operating system, including whatever was 
built on top of that nucleus. Part of the goal of these efforts 
typically was to minimize the size and complexity of the 
resulting nucleus, in the well founded belief that to do so 
would greatly increase the likelihood that the resulting soft¬ 
ware would be correctly implemented. Desires to apply pro¬ 
gram verification methods to this software have heightened 
the importance of these minimization goals, since current 
verification methods are applicable only to modest sized, 
simply structured programs. Such a minimum size, security 
oriented operating system nucleus is typically called a “se¬ 
curity kernel.” 4 

It should be noted that these efforts have taken place in 
an environment where reliable security has become of in¬ 
creasing concern. No general purpose operating system of 
a more traditional design has successfully provided a satis¬ 
factory degree of assurance for those who are seriously 
concerned about control over their data. Known flaws are 
so numerous that it is generally recognized a highly system¬ 
atic approach is required. Since it appears that kernel based 
architectures will make considerable strides in improving 
that situation, an understanding of their characteristics is 
useful. 


* This research was supported by the Advanced Research Projects Agency 
of the Department of Defense under Contract MDA 903-77-C-0211. 


There are several considerations that make a kernel based 
operating system architecture significantly different from a 
more traditional architecture developed without security as 
a major design criterion. Reliable security demands that the 
software upon which security depends be as small and sim¬ 
ple as practical. As a result, functions included in operating 
system bases to enhance performance or increase the con¬ 
venience of writing software above that base are not needed 
for security purposes and consequently not included in a 
kernel. Kernel designs would typically exclude that soft¬ 
ware despite potentially increased performance overhead or 
greater non-kernel software complexity. Naturally, the ar¬ 
chitecture is structured to avoid these costs as much as 
possible, and it appears that such penalties can generally be 
minimized. 

Conversely, the kernel must contain all security related 
software if correct enforcement is not to depend on non- 
kernel software. Therefore, as a countering influence, func¬ 
tions are forced into the kernel that may have been omitted 
from an operating system base and implemented at an outer 
level. The body of this paper characterizes the architectural 
principles that lead to these changes. Examples of the re¬ 
location of system functions in an architecture are also 
given. 


EFFECTS OF DESIGN CONSTRAINTS ON KERNEL 

ARCHITECTURES 

While kernel architectures in general exhibit many com¬ 
mon characteristics and differ substantially from traditional 
operating system architectures, they also differ significantly 
among themselves. The technical constraints concerning se¬ 
curity policy, function, hardware and performance must be 
specified before it will be possible to develop appropriate 
kernel specifications. Each of these considerations is dis¬ 
cussed in turn below. 

Security policy 

The amount of mechanism that must be included in a 
security kernel is greatly affected by the particular security 
policy (control discipline 5 or level of function 6 ) that it is 
desired to enforce. For example, the support required for 
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isolation is considerably less than what is needed for general 
support of intimate controlled sharing with domain switches 
within an individual process. 

Kernel architectures have been directed primarily at the 
goal of reliable “data security”: assuring that it is possible 
for users only to access data to which they are specifically 
entitled. Issues such as confinement or denial of service 
have generally not been directly addressed, although care 
usually has been taken to minimize such problems. Clearly, 
the more sophisticated the security policy that it is desired 
to enforce, the more kernel mechanism will likely be needed. 

Data security policies themselves may be subdivided on 
a number of criteria. What are the objects that are protected? 
An operating system kernel might support any one of the 
subsets of an extensive list of potential objects: processes, 
pages, segments, files, file subtrees, messages and/or mes¬ 
sage channels, records, fields, mini-disks, entire devices 
(tape drives, disk packs, terminals) or partitions of devices 
(blocks on a tape, windows on a crt terminal). 

Next, what is the object grain: the size of the objects 
protected. Are processes the active objects, or can a pro¬ 
cedure call within a process coincide with a domain change, 
as in Hydra. 7 Perhaps, in a system that supports a family 
tree of processes for a given user, the entire family is the 
active object, with no distinction made among processes. 
That is, all processes in the tree might be required to have 
the same access rights. This design was used in the TENEX 
operating system. 

What are the rules governing the alteration of system data 
that records the security policy? That is, what is the policy 
grain. Is control over objects and groups of objects hierar¬ 
chically distributed as in Multics, 8 or is it strictly centralized, 
as in the IBM operating systems. What is the precision with 
which one can specify access control decisions? Can a file 
be marked with the exact list of users to be given access, or 
can one only set the access mode as either public or private, 
to illustrate two extremes. 


System functions 

Apart from the question of the security policy supported 
by a kernel, a serious impact on the actual kernel design and 
implementation results from the functions and services to 
be supported by that kernel. Is it possible, for instance, to 
create and destroy objects of all types supported by the 
kernel, or do certain cases, such as mini-disks, have their 
allocation fixed at the time the kernel is assembled. To what 
degree is the number of objects of each type supported and 
protected by the kernel fully variable. How rich are the set 
of operators provided to access the kernel protected objects. 
Are sophisticated message synchronization primitives built, 
or is a simpler, less efficient facility provided, out of which 
user software must simulate the desired features. 

Type extensibility is another aspect of system function. 
Is it possible for user programs to create new objects of new 
types and have these new objects protected in a fashion 
similar to previously existing objects. This type extensibility 


first appeared in programming languages, and subsequently 
in the CAL-TSS system. 3 

Another aspect of system functionality that invariably af¬ 
fects a security kernel concerns device support. For reasons 
discussed later, most hardware designs require, for security 
to be reliably enforced, that there be additional kernel soft¬ 
ware for each device type. Therefore, the more general 
hardware base to support, the larger and potentially more 
complex a kernel can result. 

Another significant consideration affecting a security ker¬ 
nel is the convenience with which operating system software 
can be constructed on top of the primitives which are pro¬ 
vided. If certain facilities are entirely absent, selected func¬ 
tions may be impossible to provide. The existing Unix kernel 
does not permit asynchronous I/O within a user process. 9 
Other functions may be possible to build, but only in a 
relatively inconvenient way. The UCLA kernel 17 does not 
support the process hierarchies required by UNIX, although 
they can be built outside the kernel using inter-process com¬ 
munication facilities. 

In general, one wishes as little functionality in the kernel 
as is sufficient. In applying this philosophy, one should 
examine the necessary functions and determine whether a 
different, smaller and simpler set would suffice. For exam¬ 
ple, extensive real time functions, synchronization facilities, 
and interprocess communication mechanisms may be de¬ 
sired. However, the base system might be carefully designed 
only to include a small set of well chosen I/O primitives, 
augmented by a signalling call and a mechanism by which 
a user process could mark itself uninterruptible for a short 
period. The extensive set of user desired functions might 
then be constructed in application code using the simple 
base facilities. 


Hardware effects 

The hardware base on which a kernel is built can signifi¬ 
cantly impact the resulting kernel, including its size, com¬ 
plexity, overall architecture, and details of implementation. 
The richness of the hardware base that must be supported 
will have considerable effect. A multiple processor system, 
in which more than one cpu executes kernel functions, may 
require support for processor coordination within the kernel 
that single processor systems do not. Certain types of I/O 
channels may need detailed cpu software support. 

Typically, the accesses made by central processors are 
constrained by memory management units, protection keys, 
or other hardware support. That support has the character¬ 
istic that privileged cpu software can set the controls and 
then permit untrusted software to run with assurance that 
it is not possible for software to exceed those access rights 
recorded in the hardware controls. The hardware imple¬ 
mented control features associated with channel accesses to 
main memory and to secondary storage are typically much 
less convenient, or even non-existent. Channels on most 
machines use absolute addresses to access main memory, 
instead of having a virtual reference modified and controlled 




Issues in Kernel Design 


1081 


by relocation hardware. Therefore, unless some other form 
of hardware enforced protection, such as IBM’s storage 
keys, limit the channel’s access, additional kernel software 
is required to check and perhaps modify the addresses used 
by the channel in performing I/O. This task can be quite 
complex and error prone, even on IBM hardware, as re¬ 
ported by Belady and Weissman. 10 

While in some existing machines, channels’ main memory 
access is controlled by programmable hardware assists, it is 
even rarer also to find hardware support for control of access 
to the secondary storage connected to the channel. There¬ 
fore it is generally necessary for kernel software to check or 
modify the secondary storage addresses and relevant com¬ 
mands that compose channel programs. Certain machines 
such as the IBM 360/370 series have some limited hardware 
supported control over secondary storage access. There, 
changes in cylinder or track can be blocked for the duration 
of a channel program’s execution. However, such facilities 
are generally insufficient for direct use by the kernel to 
protect pages, segments or file systems. 

I/O is clearly one area where hardware characteristics 
considerably affect the security kernel, but it is not the only 
one. A simple example-, the time of day clock, on many 
machines is located in a way so that only privileged software 
can read it, despite the fact that the time of day is not 
relevant to most definitions of data security. On the PDP- 
11, both the word size and cpu arithmetic are organized on 
a 16 bit basis. Absolute addresses are larger however. I/O 
device registers (each 16 bits) have the additional bits lo¬ 
cated in additional registers, sometimes with little common¬ 
ality among devices where those bits are to be found. Some¬ 
times carry from the 16-th to 17-th bit is implemented, 
sometimes not. As a result, special software for each device 
is typically needed to check address bounds. Additional 
kernel code is required that could have been eliminated with 
judicious hardware planning. 

There are often a number of such hardware characteristics 
that considerably complicate the task of building a secure 
kernel, and that would not require significant changes to 
greatly diminish those difficulties. The previous examples 
were largely of this class. More dramatic changes to the 
hardware base could of course greatly contribute to reliable 
system security. Capability based architectures have con¬ 
siderable promise in that respect. Kernel designs such as 
those being pursued at UCLA particularly lend themselves 
to implementation in firmware. The attractive feature about 
many of these points is that their adoption should not nec¬ 
essarily imply significantly increased fabrication costs, and 
both the simplicity and reliability of software would be en¬ 
hanced. 


Performance 

The last significant issues which affect the size and com¬ 
plexity of the kernel are performance constraints. Stringent 
performance requirements as expected make it difficult to 
develop a straightforward design and implementation. For 


example, while it may be possible for scheduling decisions 
to be made through a separate user process which is only 
interrogated by the kernel, the additional process switch 
overhead to invoke the scheduler for each expected user 
process switch may impose unacceptable delays. Expensive 
domain changing usually exerts pressures that lead to larger, 
more complex kernels, since one is tempted to integrate 
such functions as scheduling into kernel code. 

The costly effects of this phenomenon are well illustrated 
by the Multics experience. 11 On its original GE 645 hardware 
base, ring crossings (domain changes) were quite expensive, 
since the existence of rings was simulated in software by 
a complete switch of segment tables and status at each ring 
crossing. As a result, performance pressures led the highly 
privileged ring zero to be composed of approximately 80,000 
lines of PL/1 code, and a number of security errors. Sub¬ 
sequent research has shown that a majority of that code can 
be removed. 4 

Even if performance requirements do not alter the func¬ 
tional specifications of a kernel, complexity may be in¬ 
creased by the need to code certain functions in a highly 
efficient manner. It may be necessary to replace a simple 
sequential search by a more complex hashing algorithm that 
uses chaining in a overflow area for collision resolution, as 
just one example. 

These issues of security policy, system function, hardware 
impact, and performance requirements all affect the size and 
complexity of a security kernel which can be built. Within 
any given set of such constraints, however, certain designs 
are far superior to others. The next section of this paper 
discusses kernel design principles that generally seem to 
apply to most of the likely sets of constraints. 


PRINCIPLES OF KERNEL DESIGN 

A number of design principles have been developed as 
guides to the design of secure systems, notably “least priv¬ 
ilege’’ and “least common mechanism’’. See Saltzer 6 or 
Popek. 8 However, they are little help without explanation 
of their architectural effect, examples of their application, 
or discussions of their effect on such operating system prob¬ 
lems as resource management or performance. 

Here we first make several general observations regarding 
kernel designs and illustrate specific tradeoffs with examples 
from existing systems. Most of these observations and illus¬ 
trations are concerned with the tasks of developing as small 
and simple a kernel as is practical, under the design con¬ 
straints discussed earlier. 

Overall system architecture 

One of the first questions concerns the relationship of the 
kernel to the rest of the operating system. There are a 
number of possibilities. In both the Multics system and a 
Unix prototype under development at the Mitre Corpora¬ 
tion, the kernel is essentially part of the user process, even 
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sharing the process address space in the Multics case. Ker¬ 
nel code and data are protected from user process software 
by a variety of mechanisms. Kernel tables are shared among 
all processes. Interrupts may occur at nearly any time, even 
while running kernel code. Typically, at that point the cur¬ 
rent process can be suspended and a process switch per¬ 
formed. Therefore kernel data and tables must be designed 
for parallel access and modification. 

An alternate approach, examined at UCLA, more com¬ 
pletely separates the kernel from all other software. As in 
the RC 4000 kernel, the UCLA security kernel is not part 
of any process. It is run in its own address space, and is 
better thought of as a hardware extension. That is, each 
software implemented primitive instruction provides some 
security relevant function, and runs from invocation to com¬ 
pletion without interruption. Functionally it acts as one 
hardware instruction. Therefore, a process state that indi¬ 
cates running in a powerful mode is never saved or restored, 
and parallelism complexities are considerably reduced or 
eliminated. While such an approach may reduce kernel com¬ 
plexity, it requires care in the overall architecture if ade¬ 
quate functionality is to be obtained. The values of each of 
these approaches are discussed in more detail elsewhere. 12 


Resource pools 

One of the most effective ways to reduce the size and 
complexity of an operating system kernel is to remove, as 
much as possible, all of the resource management functions 
for a particular type of resource and relegate those functions 
to distrusted code. The degree to which this goal can be 
attained, of course, will vary from system to system, and 
from one object type to another within a given system. 

There are three significant aspects to resource handling in 
operating systems that are subject to removal from an op¬ 
erating system kernel and supportable by user software. 
They are type integrity, management policy, and naming. 
Each is discussed in turn below. 


Type integrity 

The most extreme action possible is to remove the entire 
resource as well as all of its supporting software from the 
base level of the system. As a result, the kernel does not 
provide protection for these resources as such. Any structure 
which is needed to insure the type integrity of the resource 
is the responsibility of the user software. The only protection 
enforced by the kernel is that provided for the kernel rec¬ 
ognized object in which the user implemented resources are 
placed: segments, pages, disk blocks and the like. 

What this strategy generally means is that a single com¬ 
mon pool of resources, usually maintained for all processes 
by the operating system, has now been broken up into a 
number of smaller subpools, each managed independently. 
This approach is illustrated by the I/O buffer management 


in OS/360. Buffer integrity is not assured, since whatever 
pointer the user supplies in certain SVCs is used by the 
operating system so long as the indicated transfer involves 
that user’s core. 

The tradeoffs involved in pursuing this design are straight¬ 
forward. By moving all of the resource handling out of a 
kernel, including the maintenance of the structure or type 
of the resource, the kernel is further simplified, sometimes 
considerably. The performance cost is clear from elementary 
queuing theory. Separate subpools are in general more 
poorly utilized than the same number of resources in a single 
common pool, because some subpools may contain idle re¬ 
sources while others are exhausted and unable to fulfill 
requests directed at them. Alternately, additional resources 
will be required to maintain the same level of service. A 
designer must decide what the cost of additional memory or 
other resources is, or how much delay in response can be 
tolerated in return for what is likely to be a more secure 
system. 

There are many resources that an operating system typi¬ 
cally manages that are not so obvious as I/O buffers. Name 
spaces are a good example. There are a number of cases 
where unique names are needed for convenience in com¬ 
munication, like socket numbers in the ARPA net, which 
are usually assigned and reclaimed by the operating system. 
However, they could be preallocated to various domains, 
with each domain expected to do its own management. A 
kernel, using lower level names, can assure that messages 
are properly delivered to the appropriate domains in the 
ARPA net example, but need not be concerned with man¬ 
agement of sockets. 13 

Resource management 

In those cases where it is not possible to completely ex¬ 
clude support of a resource type from the kernel, it may be 
possible to remove the management of the resource, but 
leave that portion of the software responsible for the integ¬ 
rity of the resource objects. That is, the mechanisms which 
actually operate on the resource are part of the kernel, but 
the policy software which decides which operations should 
be performed, and under what circumstances, is distrusted, 
and part of user processes. This approach, which leaves 
type integrity as a kernel responsibility but resource man¬ 
agement relegated to user code, is essentially the distinction 
made by Wulf between mechanism and policy. 7 

A good example of the application of this approach occurs 
in the UCLA kernel architecture. 14 There, process sched¬ 
uling as well as the management of main memory is the sole 
responsibility of one particular user level process, containing 
distrusted code. It issues kernel calls to switch processes, 
initiate page swaps, and so forth. While this scheduling 
process therefore can move around the objects that it has 
the responsibility to manage, it cannot examine any of their 
contents. This method supports usual functionality require¬ 
ments and permits sophisticated scheduling, while it consid¬ 
erably simplifies the kernel. However, certain confinement 
problems are not solved, as discussed later. 
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Naming 

As pointed out by Janson, 15 a considerable amount of 
code in an operating system is devoted to managing the 
names of resources, in addition to the management of the 
resources themselves. A resource may in fact be known by 
several names, at differing levels of abstraction. In addition 
to enforcing conventions such as uniqueness of names, the 
mapping of one name to another must also be done. An 
example of this mapping can be seen in the handling of 
segments in Multics. Character string names for segments 
are used by the file system software (and by users) to specify 
a segment (analogous to a file). Segment numbers, indexes 
into a “Known Segment Table,” are used by running pro¬ 
grams to refer to active segments. The Known Segment 
Table of that user contains the physical address of the page 
table for the corresponding segment. Thus there are three 
different name spaces used to refer to segment resources, 
and mappings among these spaces must be maintained cor¬ 
rectly. 

The more of this name management that can be removed 
from the kernel, the simpler the resulting kernel software. 
There are several ways to do so. One need not fully support 
in the kernel each of the name mapping levels. Several of 
them, including their mapping software, could be built as 
part of user domains. This approach generally amounts to 
partitioning the higher level name spaces, with separate par¬ 
titions managed by separate processes. However, such a 
design requires care if controlled sharing is supported by the 
architecture. User software certainly needs to know the 
names maintained by other user software in order to coor¬ 
dinate references to shared objects. The coordination can 
be accomplished if each name maintenance package follows 
the same conventions, and messages are used to exchange 
relevant information. 

Of course, it is true that in any system some conventions 
must be externally agreed upon, and some information ex¬ 
ternally exchanged between people, in order to get coordi¬ 
nation started, even if it is only where (in what directory 
perhaps) to look to find relevant information. Here it is 
being suggested that much of this coordination and conven¬ 
tion handling need not be part of the security enforcement 
of an operating system. The one potential limitation in this 
viewpoint, concerning so called Trojan horses, is discussed 
in a later section. 

One should note that it is possible to move only a portion 
of the name support out of the kernel, too. One might par¬ 
tition the name space, allocating each partition to a different 
user process, and then have the kernel merely enforce that 
partition. Each user would do his own name management 
within a given partition. This view is similar to that of type 
integrity, discussed earlier, with the resource being the 
names, and the operations implemented in the kernel which 
operate on names being very limited. 

DECOMPOSITION OF SECURE CODE 

Up until this point we have maintained the fiction that the 
appropriate system architecture for highly reliable security 


placed all security relevant code in the core of the operating 
system, the kernel, running on the bare hardware. The rest 
of the system software is encapsulated within user pro¬ 
cesses, running under the control of the kernel, and making 
calls to it. 

Trusted processes 

However, this structure is not the best, as recognized by 
work at M.I.T., 11 UCLA, 12 SRI, 16 as well as in specifications 
for the military network SATIN-IV. Rather, there are a 
number of security relevant functions that can profitably be 
moved out of the kernel and placed in so called “trusted 
processes”. These processes architecturally are the same as 
user processes, except that the information they pass to the 
kernel is used by the kernel to take security relevant actions. 

A good example of such a trusted process appears both 
in the Multics system at M.I.T. as well as in the UCLA 
Secure Unix development. One process has responsibility 
for handling free terminals, logging and authenticating users, 
and starting up the appropriate software for them. The Mul¬ 
tics logger and the UCLA initiator both run as standard 
processes, except that the Multics ring zero software and 
the UCLA kernel both accept the authentication decision 
made by that process, and make security relevant decisions 
using that information. 

Levels of kernels 

One sometimes finds that the software composing these 
trusted processes can be segregated into security relevant 
and nonsecurity relevant parts. That is, the trusted pro¬ 
cesses can often themselves be structured so as to be com¬ 
posed of a kernel and other, distrusted software. 

The degree to which this segregation is successful can be 
increased if a few redundant checks are made. For example 
one might build a sophisticated hashing algorithm with 
clever collision resolution for password checks, but after the 
appropriate entry in the hashing table has supposedly been 
found by distrusted code, the table index can be given to a 
trusted subroutine to make the actual check and pass infor¬ 
mation to the operating system kernel. In VM/370, channel 
program translation is followed by a security check of the 
resulting code. 

The net result of this segregation is a system composed 
of levels of kernels. Each kernel depends, for its correct 
operation, only on the correctness of its own implementa¬ 
tion, together with that of the lower level kernels that pro¬ 
vide security relevant functions. In this way, the necessity 
that there be security relevant, trusted code at various levels 
in a large system is met, while at the same time the total 
amount of code upon which the overall system’s security 
depends has been minimized. Note that this viewpoint dif¬ 
fers markedly from that taken at SRI, in which it is argued 
that the entire operating system design and implementation 
should be certified or verified. 16 

A more powerful example of the utility of a second level 
kernel concerns name mapping and file systems. There, 
conditions occur under which the user needs to refer to 
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system supported objects by names that are from a different 
name space than that supported by the kernel. For example, 
as already mentioned, the kernel may maintain segment 
numbers, while the user will invariably wish to refer to those 
segments by character string names, especially when setting 
access control information, but perhaps also when stating 
which segment to read or modify. Because of these usage 
patterns, mapping between name spaces may be highly se¬ 
curity relevant. File systems provide an important case of 
this problem. In the UCLA Unix system, file management 
is done by a user level process. 17 The kernel of that file 
manager is responsible for elementary directory manage¬ 
ment, and other distrusted software in the process can take 
care of positioning pointers within files, scheduling I/O op¬ 
erations and maintaining resource allocation limits. The di¬ 
rectory management consists largely of mapping string 
names to disk block names supported by the kernel, handling 
allocation matters, and checking and maintaining protection 
data. 

This multiple level kernel architecture can yield significant 
advantages in the design and implementation of the base 
level kernel. If applied in conjunction with other design 
principles discussed above, the base level kernel can become 
little more than an implementor of abstract types. It makes 
segments, processes, messages, and so forth out of hardware 
memory, status registers, and the like. For some systems, 
this simplification of the base level kernel may permit further 
improvement. It may be possible to implement each kernel 
call (including the interrupt handler, which is just a hardware 
forced call) as uninterruptible code, without losing any rel¬ 
evant performance characteristics. This design virtually 
eliminates any concern for parallelism in the security rele¬ 
vant code, and therefore enhances its reliability. If the kernel 
is simple enough, it is a candidate for implementation in 
microcode. A packet switch communications processor that 
hosts no user programming is an example of an application 
that might only require a kernel simple enough for such an 
implementation. 

A final note on levels of kernels concerns the updating of 
the protection data used by trusted software to make access 
control decisions. If a user at a terminal is to have confi¬ 
dence in the security enforced by a system, he must be able 
to view and change the system’s protection data in a highly 
reliable way. He needs a secure channel between his ter¬ 
minal and kernel code, without any distrusted software in¬ 
tervening, even to manage character handling. One easy 
method to implement that channel is to provide a simple 
facility by which the user can switch his terminal to a trusted 
process and back. In this way he need not be concerned 
with any software running in his own process(es). While this 
solution, unless extended, suffers from the inability to per¬ 
mit program generated changes in protection data, such ac¬ 
tions do not seem to occur often in practice anyway, except 
as a way of compensating for some other lack in function¬ 
ality or flexibility of the protection controls. 

INTERNAL KERNEL ARCHITECTURE 

Much of the discussion up to this point has concerned the 
architectural place of a kernel in the larger system architec¬ 


ture, and the effect of kernel goals on that structure. Natu¬ 
rally, the internal structure of a kernel is also important. In 
general, the task of constructing a reliable kernel is little 
different from that of developing other highly reliable soft¬ 
ware. However, there are considerations peculiar to this 
application which deserve mention, including hardware se¬ 
lection, parallelism, and the effect of finite resource limits 
on attempts to use abstract type methods for organizing 
kernel code. 


Hardware selection 

Since hardware is the zero-th level in a structured, layered 
software architecture, unnecessary complexity in that inter¬ 
face is as undesirable as complexities in the specification of 
any other layer in the software, for the usual, good reasons. 
Therefore, hardware selection requires care, as illustrated 
earlier. It is worth noting that since the kernel software has 
already been minimized, the hardware impact on what re¬ 
mains can be substantial. In the UCLA kernel for example, 
approximately 40 percent of the code is device drivers, much 
of which could be eliminated if address relocation applied 
to device transfers. 


Parallelism 

While there may be some argument over whether algo¬ 
rithms that are expressed as parallel computations or strictly 
sequential code are inherently easier to understand (to some 
degree it certainly depends on the algorithm), it seems rather 
clear that certification or verification of sequential code is 
presently less difficult. That is the reason for specifying 
physical or logical uninterruptibility in the software archi¬ 
tecture. This goal is, of course, one that has been followed 
at higher levels in systems since the early days of multipro¬ 
gramming. 

Abstract type structures 

The concept of using extended types (such as illustrated 
by Simula classes, 18 CLU clusters, 19 Alphard forms, 7 or 
Euclid modules 20 ) to organize software is quite valuable. 
However, Janson 15 points out that most efforts to employ 
strict type extension methods in programming are done with 
the underlying assumption of unlimited quantities of re¬ 
sources available. In the internal structure of an operating 
system or its kernel, this assumption is invalid, of course. 
One of the most obvious cases where finite resource limits 
interact with system structure occurs in virtual memory 
systems, potentially resulting in a structural circularity. 
Main memory is finite, so system software is constructed to 
give the illusion of a larger address space by sensing page 
faults and performing necessary I/Os in a manner invisible 
to the process involved. However, one may well wish to run 
this virtual memory software as a process like other pro¬ 
cesses. Further, the process management software can ben¬ 
efit from being written with the assumption that a large, 
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virtual address space is available. With the obvious circular 
dependency in mind, it is not clear whether process types 
or virtual memory types should be the lower level, with the 
other built on top. Reed 21 outlines a method of defining 
multiple levels of types to break the cycle. Lower levels 
implement a small fixed number or amount of the resource, 
using small amounts of resources in so doing. Higher levels 
reimplement the resource in large numbers or amounts, by 
multiplexing the lower level ones. In medium sized systems 
such as UCLA Unix, such a multiple tiered structure is not 
necessary. 


CONFINEMENT 

The problem of constructing a system so that an arbitrary 
program can be run in a fashion that assures the program 
cannot leak any of the information that it contains, is in 
practice unsolved. While clearly solvable in principle, 22-24 
no actual solutions have been designed or implemented for 
real systems. 

Importance 

Some have argued instead that confinement is not a seri¬ 
ous issue in practical applications. While this viewpoint is 
clearly reasonable when taken in the context of the current 
absence of reliable data security, several experiments with 
the Multics system have illustrated how real the problem 
may be. It was found possible in principle to drive a user 
terminal through interprocess communication performed by 
modulation of paging behavior in one process and the sens¬ 
ing of that modulation by the other. In a separate experi¬ 
ment, a similar bandwidth was discovered through use of 
the directory system. One process repeatedly made and 
deleted entries in a directory. This action changed the size 
of that directory, a value recorded in its parent directory. A 
second process read the size of that parent directory. In this 
particular test, the first process was not supposed to be able 
to communicate to the second. 11 

These two examples are instructive, for they illustrate a 
timing dependent and timing independent channel, respec¬ 
tively, both involving resource management. The bandwidth 
is enough to be of concern both to the military (one job 
might be a “Trojan horse” running with top secret classifi¬ 
cation, and the other running unclassified) as well as to 
industry (consider disgruntled programmers). 

From a practical point of view, virtually all the channels 
mentioned by Lampson 23 or illustrated above are related to 
the allocation or management of resources, and the changes 
in system data that occur as a result of resource allocation 
decisions. 

Storage and timing channels 

Millen et al. 24 partition these channels into storage chan¬ 
nels and timing channels. Storage channels are those which 
permit the passing of information through direct modifica¬ 


tion and reading of cells in the computer memory. Timing 
channels are those which permit one user to vary the real 
time rate at which another receives service, to a degree that 
the second user can sense. As the examples earlier show, 
the bandwidth of both types of channels can be significant. 

It appears, however, that all resource storage channels 
can be changed into timing channels, and, on a large system, 
the bandwidth of the timing channels can be severely lim¬ 
ited. Further, it appears that these actions can be taken 
without losing the values of multiprogramming, which is the 
primary cause of the problem. Resource storage channels 
can in general be changed to timing dependent channels in 
the following way. First, it appears that all direct reads and 
writes of system storage, such as that illustrated by the 
Multics file system problem, can be blocked by proper def¬ 
initions of domains and the association of status data in the 
appropriate domains. What remains are resource allocation 
requests and status notifications. The use of resource avail¬ 
ability as a channel can then be prevented merely by block¬ 
ing the process requesting the resource until it is available. 
To be practical, this replacement of resource status replies 
by delays should be accompanied by the use of some form 
of a deadlock detection or prevention algorithm to insure 
that the system operates efficiently. That code need not 
necessarily be implemented in a security relevant way, as 
shown by the discussion of scheduling below. 

Some believe that the bandwidth of timing channels can 
be limited by the scheduling characteristics that can be in¬ 
troduced in a large system. However, the existence of a 
timing channel is in principle more difficult to identify than 
a storage channel, since in the latter case there are explicit 
cells which are being accessed, even if interpretively through 
system code. While one might argue that if a process has 
only virtual time available to it, the timing channels are 
blocked, this viewpoint is probably not practical, given how 
many real time devices (therefore, clocks) actually exist on 
real systems. Nevertheless, once the bandwidth is limited 
to an amount comparable to that expected external to the 
computer system, the problem has in effect been solved. 

Timing independent scheduling channels 

While the preceding paragraphs may suggest a general 
approach to confinement, considerable care in the system 
architecture and implementation is required, as illustrated 
below. Earlier it was pointed out that functions such as the 
scheduling policy for resource allocation could be done out¬ 
side a kernel. With respect to confinement, however, this 
approach creates difficulties. A reasonable scheduler must 
receive considerable information if it is to perform effective 
optimization. Since it can transmit information by ordering 
the completion of user requests, there is a clear potential for 
covert communication whether or not the scheduler had 
been written to cooperate. 

Storage channels can in large degree be blocked however 
if the scheduler is designed as an information sink. This 
action can be taken if each user is permitted to submit only 
one request at a time, and wait for its completion before 
submitting another. Device completion status can be re- 
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turned via the kernel. Studies have shown that, for a rea¬ 
sonable number of jobs on a multiprogramming system, 
these restrictions are generally not ones with significant 
impacts on performance. 10 

However, such restrictions do not prevent collusion 
among user processes from providing timing independent 
channels. That is, a group of cooperating, legally commu¬ 
nicating processes may be able to communicate with another 
such group, with which they are not permitted, even in the 
face of a scheduler designed to be an information sink. 
Whether or not this is possible depends on scheduler char¬ 
acteristics. Whenever it is possible for one process, by its 
own resource utilization actions, to affect the relative order 
in which members of other groups of processes are served, 
communication is possible. 

That is, let processes A and B be one group, X and Y the 
other. Inter-group communication is not to be permitted. 
(They may be Ford and G.M. programs). Suppose that A 
(or B) increases its cpu to I/O ratio so much that the system’s 
adaptive scheduler stops favoring cpu bound processes and 
begins favoring I/O bound jobs. X is cpu bound, Y is I/O 
bound. The order in which X and Y are run was changed by 
A or B. X and Y can easily determine the relative order in 
which they are run since they can legally communicate. 
Thus the A,B group has sent one bit to the X,Y group. 
Clearly this mechanism serves as a basis for extended bi¬ 
directional communication. 

Many of the interesting scheduling algorithms that are 
used in practice have the characteristic outlined above, and 
in fact those that don’t seem to ignore precisely that kind of 
information concerning user behavior upon which meaning¬ 
ful optimization is based. However, not all useful algorithms 
permit collusion. Round robin does not. The simple, limited 
algorithm implemented in the new Mitre Unix kernel, in 
which each process can only set its own priority, and the 
kernel merely runs that user process with the highest prior¬ 
ity, also blocks collusion. 

If confinement is felt to be important, considerable audit¬ 
ing of resource management mechanisms and policy will be 
necessary to block or satisfactorily limit the bandwidth of 
these kinds of channels. 

CONCLUSION 

We have attempted to distill the knowledge gained from 
various security kernel research efforts into a form that will 
be useful in guiding others who wish to develop highly se¬ 
cure, reliable system bases. It is also hoped that these per¬ 
spectives are applicable to other environments than operat¬ 
ing systems, especially such higher level software projects 
as message systems and data management, where privacy 
considerations are especially sensitive. 
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INTRODUCTION 

This paper considers the problem of attaining computer sys¬ 
tems and applications programs that are both highly secure 
and highly reliable. It contrasts two current alternative ap¬ 
proaches, one remedial, the other preventive. A remedial 
approach is outlined based on a classification of software 
security violations suggested by Bisbey, Carlstedt, and Hol- 
lingworth at 1ST This remedial analysis is then related to a 
preventive approach, illustrated here by the formal SRI Hi¬ 
erarchical Development Methodology. Evaluation of system 
security is then considered by combining concepts from the 
preventive and remedial approaches. This combination of 
techniques seems to have significant potential in the attain¬ 
ment and evaluation of computer system security. Illustra¬ 
tions are given for three types of systems, the first two being 
systems explicitly designed with security in mind, and the 
first of those being designed according to a formal method¬ 
ology. The first system is the SRI design for a Provably 
Secure Operating System (PSOS), the second is Multics, 
and the third is UNIX. (The reader familiar with security 
may wish to skim the next two sections.) 

BACKGROUND 

Computer systems and applications are increasingly being 
called on to provide reliable security, although most existing 
commercial computer systems are incapable of supporting 
various security-critical applications. Until recently, the task 
of obtaining highly secure systems has typically been con¬ 
sidered to be very difficult, and avoided on the grounds that 
good solutions would be very expensive. There has been 
little deep understanding of either how to develop secure 
systems and applications, or how to assess security; how¬ 
ever, there have been efforts to detect insecurity. The in¬ 
teractions between security on the one hand and reliability 
and survivability on the other have also been largely ignored, 
as have the interactions with dynamic auditing of security. 
The term “system defensiveness” is used here to imply 
security, reliability, survivability, and the auditability of any 
events that might compromise these aspects of computer 
behavior. 

Recently, the situation has been improving (e.g., see the 
survey in Shankar 1 ). Two approaches to attaining better 
security are considered here, one basically remedial, the 


other basically preventive. The first approach involves as¬ 
sessing the security of computer systems (particularly ex¬ 
isting ones) and attempting to patch around any security 
flaws thereby uncovered. A variety of techniques have been 
used to detect flaws that reduce security. These include 
searching code for certain patterns of program statements 
corresponding to classes of would-be security flaws (e.g., 
Bisbey 2 ), as well as more traditional experimental penetra¬ 
tion attempts (e.g., McPhee 3 ). The second approach in¬ 
volves designing new systems that are intrinsically secure 
and whose security can in some way be convincingly estab¬ 
lished. This approach might rely on a suitable design meth¬ 
odology (e.g., Robinson et al. 4 ), combining formal specifi¬ 
cations (e.g., Pamas, 5 Robinson et al. 4 ), suitable design 
structures, the use of suitable programming languages (e.g., 
see Lohr 6 ), and supporting tools. Examples of such ap¬ 
proaches that also support formal verification are found in 
Robinson et al. 4 and Good. 7 

With respect to the remedial approach, experience in at¬ 
tempting to penetrate allegedly secure systems leads to sev¬ 
eral observations. First, penetrators do not generally have 
to work very hard to find major security flaws in traditionally 
developed systems. Second, patches that attempt to remove 
such flaws are themselves often flawed. Third, this approach 
is intrinsically limited because attempts to characterize the 
still undetected security flaws are speculative, at best. In 
general, attempting to retrofit security into a basically in¬ 
secure system is of limited effectiveness, and inevitably 
leaves much in doubt. 

In the long run, use of the preventive approach is likely 
to be significantly more productive. This approach should 
proceed with security as a fundamental design goal from the 
outset. It may include explicit statements of requirements 
and formal specifications of the design. It should make use 
of a modern programming language, and might employ for¬ 
mal proofs of significant properties regarding system behav¬ 
ior, such as security (e.g., see Neumann et al. 8 ). On the 
basis of recent research, it is now possible to assess the 
security of a precisely specified design, prior to implemen¬ 
tation, and then subsequently to assess the security provided 
by the implementation. 

The conclusions of Glaseman et al. 9 are extremely pessi¬ 
mistic, at least for the near future, with respect to obtaining 
quantitative assessments of the cost (risk) associated with 
various security violations. Similar conclusions are drawn 
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in the present writing, with regard to attaining meaningful 
measures of security in conventionally designed systems. 
However, there are some hopes for attaining secure sys¬ 
tems. First, it is helpful to assess insecurity—although it is 
not very reassuring to know just that another security flaw 
has been found. Second, it is possible to consider relative 
measures of insecurity for different classes of would-be vi¬ 
olators such as skilled system programmers, data processing 
supervisors, system operators, skilled users, and casual 
users. Some security flaws may provide threats only from 
users in certain of these classes. Knowledge of the would- 
be violators and the degree to which they are trusted is thus 
fundamental to any evaluation of risk. Third, there is some 
short-term hope in a combination of the remedial and pre¬ 
ventive approaches in redesigning an existing system so that 
its security-relevant portions can be isolated into a small 
and cleanly defined collection of system programs (a “se¬ 
curity kernel” plus a few trusted processes, jointly respon¬ 
sible for the security of the system). However, the effec¬ 
tiveness of this combination can be reduced by restrictions 
imposed on the required redesign, such as the need to main¬ 
tain compatibility with the original (insecure) system inter¬ 
face. It also may confuse policy (e.g., security strategies) 
with mechanism (e.g., the protection mechanisms upon 
which the policy is implemented), potentially embedding 
high-level policy issues inflexibly into low-level mecha¬ 
nisms. In any case, it is felt that the preventive approach 
must be involved whenever new systems are to provide 
substantially more security than existing systems. In partic¬ 
ular, the use of a formal development methodology for de¬ 
sign and implementation can provide a powerful basis for 
intrinsic and quantitatively assessable security. 

SYSTEM DEFENSES 

Defensiveness is considered here as a generalized notion 
of security, and implies the nonexistence of inappropriate 
behavior—insofar as possible. For present purposes, defen¬ 
siveness consists of related and partially overlapping com¬ 
ponents, namely security, reliability, availability (including 
recoverability), and auditability. 

(1) Security as used here involves the protection of the 
system, its applications, and the shared resources 
from misuse. The context considered is purposely a 
broad one. It includes the notions of preventing un¬ 
authorized acquisition and unauthorized modification 
of information, that is, assuring “confidentiality” and 
“integrity,” respectively. It is intended to cover both 
system security and user data security. [In usage by 
MITRE, viz., Biba 10 “(data) integrity” is a precise 
formal dual of multilevel “(data) security” (i.e., con¬ 
fidentiality), the former referring to the writing of data, 
the latter to the reading of data.] In usage here, se¬ 
curity also includes the prevention of denial of service 
(maintaining the integrity of resources) and the pre¬ 
vention (where possible, or otherwise the limitation) 
of information leakage through clandestine channels 


(such as signalling through shared resources or 
through shared timing channels). (See Lampson, 11 
Lipner, 12 Neumann et al. 8 ) In certain applications the 
notion of security may refer to or include multilevel 
(e.g., military) security with levels such as TOP SE¬ 
CRET, SECRET, etc. (See Bell and LaPadula, 13 
Feiertag et al. 14 ) 

(2) Reliability, Availability, and Recovery involve (re¬ 
spectively) the correctness of system security (includ¬ 
ing data integrity) and other system functions; the 
maintenance of a secure running system; and the res¬ 
toration of a secure running system following any 
error, accident, willful damage, or disaster that caused 
a loss of service or security. 

(3) Auditability involves the monitoring of the continued 
existence of system security and reliability, including 
the detection of anomalous or potentially threatening 
behavior. 

Continuous maintenance of security depends on appropriate 
system hardware and software, including concepts of fault- 
tolerant architecture and reliable software. It also depends 
on an appropriate operational environment. It requires 
global planning and management. The implications of an 
unreliable system on security are particularly insidious, and 
thus must be considered along with the more typical consid¬ 
erations of security. 

It is a difficult task to assure dynamically the continuous 
absence of all would-be security violations, e.g., to audit 
misuse of an intrinsically insecure system. It seems much 
more fruitful to work on the design of better systems, and 
then to develop auditing procedures as an integral part of 
the design. (For example, auditing should obey security 
policies wherever possible.) However, once a breach of 
system defenses is discovered, it should be fixed as rapidly 
as possible. In the sense that the effects of a security vio¬ 
lation may already have leaked out, it may not be possible 
to provide dynamic recovery in the strict sense. Thus au¬ 
diting is important to help limit the propagation. Good design 
should anticipate the needs of both auditing and recovery, 
with auditing that strongly limits the propagation effects of 
detected violations and that simplifies recovery therefrom. 

Given an existing system not designed to be secure, the 
elimination of potential threats becomes more and more 
difficult after the first few threats are removed. The process 
is time-consuming, frustrating, unpredictable, and ultimately 
limited by the knowledge that although fewer threats are 
being discovered, many more may still remain undiscovered. 
It is particularly perverse that the changes necessary to 
remove discovered threats may themselves lead to new vi¬ 
olations, particularly in conventional systems. 

Given an arbitrary system, it would be useful to have 
some assessment of the penetrability of the system. How¬ 
ever, even the most innocent-looking flaw may in fact be a 
hole in the dike that leads to total inundation. In general, 
the risks from system violations can best be avoided by a 
combination of good design, good implementation, good op¬ 
erations, and good management, all done with defensiveness 
in mind. A combination of testing and penetration studies, 
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and possibly some formal verification is recommended, 
commensurate with the desired system requirements, the 
expected difficulty of penetration, and the cost of having 
violations. Formal verification for the most security-critical 
portions of the design and its implementation is expected to 
be cost-effective in the near future. Recent techniques for 
verifying design properties, independent of any implemen¬ 
tations, are particularly promising. 

The remainder of this paper is organized as follows. First, 
a categorization of various typical system violations is given, 
based on the work of Bisbey, Carlstedt, and Hollingworth 
at ISI. This provides the framework for a remedial analysis 
of any particular system, A collection of preventive criteria 
is then given whose presence in the design, implementation, 
evolution, and operation of new computer systems may re¬ 
sult in systems that are far more defensive than the conven¬ 
tional systems available today. 

It is felt that these categories of violations and criteria for 
defensiveness should be very useful for evaluating the de¬ 
velopment of new systems and applications, as well as ex¬ 
isting ones. They are used here first for a consideration of 
the design of PSOS (a Provably Secure Operating System), 
intended to be demonstrably secure. Multics and UNIX are 
then considered as two (extreme) examples of existing sys¬ 
tems. It is seen that a system developed according to the 
preventive criteria intrinsically avoids many of the flaws 
indicated by the remedial approach. 

VIOLATIONS OF SYSTEM DEFENSES 

Nielsen et al. 15 have analyzed over 300 cases of computer 
misuse (from the files of Donn Parker) and have identified 
seven types of violations to system defenses, including cases 
of disaster, accidents, hoaxes, threats and extortions. (De¬ 
fensiveness is therein called “integrity.”) However, system 
violations are involved in the preponderance of cases (e.g., 
undesired acquisition, modification, insertion, or destruction 
of data, along with various other forms of system penetra¬ 
tion). Some of the cases involve undesired denial of service. 
The sophistication of leakage through clandestine channels 
was apparently unnecessary, given the ease of penetration 
by simpler means. Nevertheless, in deference to Murphy’s 
Law (“If it can happen, it will”), all plausible violations 
should be anticipated, and covered by various safeguards 
whenever the threats are deemed significant. 

Classes of system penetrations 

Bisbey, Carlstedt, and Hollingworth at the USC Infor¬ 
mation Sciences Institute (ISI) have studied techniques for 
identifying potentially vulnerable sections of code, including 
some techniques that could be carried out automatically. In 
so doing, they identified 10 categories of system flaws, each 
capable of producing security violations. 2 These categories 
have evolved over the last few years, with each violation 
observed at ISI fitting into at least one of these categories. 
However, there is no pretense that these categories are 


either canonical or complete. (Although they were con¬ 
ceived primarily to be applicable to software, most of them 
are in fact also applicable to hardware.) 

Two of the 10 ISI categories are closely related and have 
been lumped together here. For present purposes, the re¬ 
sulting nine topical categories are grouped into four generic 
groups of flaws. Each has both design and implementation 
aspects. 

(A) improper protection (initialization and enforcement); 

(B) improper validation; 

(C) improper synchronization; 

(D) improper choice of operation or operand. 

The nine topical categories can be associated with these 
groups as follows. 

PROTECTION (initialization and enforcement): 

(1) improper choice of initial protection domain; 

(2) improper isolation of implementation detail; 

(3) improper change (e.g., a value or condition changing 
between its time of validation and its time of use); 

(4) improper naming; 

(5) improper (incomplete) deallocation or deletion; 

VALIDATION: 

(6) improper validation; 

SEQUENCING: 

(7) improper indivisibility; 

(8) improper sequencing; 

OPERATION CHOICE: 

(9) improper operation or operand selection. 

These categories are illustrated in Table I. They cover many 
different security violations, including undesired reading, 
writing, and deleting, undesired denial of service, and to 
some extent undesired leakage through clandestine infor¬ 
mation channels. For example, visibility of implementation 
detail (category 2) often leads to potential leakage channels. 
However, leakage channels also exist in systems that are 
otherwise secure, because of the natural visibility of elapsed 
time. 

It is not claimed that the four generic categories are or¬ 
thogonal. Nevertheless, they provide very useful typical 
cases. Within the first generic category (“improper protec¬ 
tion”), the five topical categories exhibit considerable 
overlap. (Category 1 deals with initialization, while categories 
2-5 provide paradigms of improper enforcement.) In fact, 
there exists a sequence of flaws that exhibits an ordering of 
(1)=>(2):^>(3)=>(4)^(5), as follows. An incorrect choice of 
protection partition (1) can result in the inability to isolate 
the implementation of an abstraction from the use of that 
abstraction (2). Such inability can result in a value presented 
at the time of call (and expected subsequently to remain 
constant) being changed during execution (3). Such a flaw 
can result in a naming problem (4), in which two different 
paths to the (apparently) same data can give different results. 
Finally, a naming problem may create a residue problem (5), 
e.g., if a local symbolic name is not unbound when the 
object is deleted with reference to its global name—or vice 
versa. (Since each category may reappear at different levels 
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TABLE I.—Categories of Protection Flaws (and examples) (Based on 
Bisbey, Carlstedt, Hollingworth at ISI) 

1. Incorrect choice of protection domain or security partition 

(a security-critical function manipulating critical data directly accessible 
to the user; incorrect initial assignment of security or integrity level at 
system generation, configuration, or initialization) 

2. Exposed representations or implementation detail 

(bypassing an abstraction, e.g., direct manipulation of a hidden data struc¬ 
ture such as unmediated user modification of a directory entry; user use 
of an absolute 1-0 address; note that the visibility of timing information 
provides a generic leakage channel, e.g., drawing inferences from page 
fault activity) 

3. Inconsistency of data over time 

(noninvariance of parameters, e.g., change in value of a parameter in a 
call by reference; change in a file accessible to different processes, e.g., 
in an improperly protected, shared process directory) 

4. Naming problems 

(aliasing, e.g., two distinct names for the same object not being treated 
identically; ambiguity resulting from use of the same local name for two 
distinct objects [which may also involve a residue problem, e.g., if name 
persists in some system table]) 

5. Residues in allocation and deallocation 

(incomplete deletion, revocation, or deallocation, e.g., such that an ap¬ 
parently deleted value is still accessible—in core, disk, archive store, etc.; 
incomplete cleanup on abort; ignoring terminal hangup). 

6. Nonvalidation of critical conditions and operands 

(invalid or unconstrained parameters such as an out-of-bounds virtual 
address or absolute 1-0 address; lack of strong type checking, e.g., a 
pointer to a structure of the wrong type; absence of quota limit stops such 
as bounds on queue sizes or number of processes, with overflows resulting 
in possible system or user crashes) 

7. Indivisibility problems (in multiprogramming) 

(interrupted atomic operations, e.g., incomplete interrupt handling [quit 
during login resulting in partial success, or in perpetual lockup of inter¬ 
locked data]; faulty read-alter-rewrite in hardware) 

8. Serialization problems (in multiprogramming, multiprocessing) 

(incorrect sequencing [e.g., wrong order], improper isolation of atomic 
operations from one another [e.g., reading during writing, or concurrency 
among different directory commands on the same directory]; critical race 
conditions in implementation; deadlocks and deadly embraces) 

9. Incorrect choice of operation or operand 

(use of the wrong function, producing incorrect results; use of an unfair 
scheduling algorithm, producing correct results for each scheduled proc¬ 
ess, but denying service completely to certain users) 


in a hierarchical design, other such orderings may also 
exist.) Thus the most primitive descriptor of these protection 
flaws is something like “improper domain selection and 
enforcement.” 

There is some overlap between the two topical categories 
of sequencing flaws. The indivisibility and serialization cat¬ 
egories, (7) and (8), respectively, are related when consid¬ 
ered at different levels of abstraction, since (logically) 
atomic operations appearing at one level may in fact be 
implemented out of (logically) atomic operations at a lower 
level, with appropriate interlocks. That is, a serialization 
problem at one level may apparently manifest itself as an 
indivisibility problem at the next higher level. 

Validation problems usually involve the omission of a 
check on argument validity or on quota limits. The incon¬ 
sistency of data over time (category 3 above) might con¬ 
ceivably be viewed as a validation problem. (This is the 
time-of-check-to-time-of-use flaw, or “TOCTTOU,” con¬ 
sidered by McPhee, Weissman, and others.) However, this 
view is somewhat like locking the barn door after the horse 
has escaped: the primitive flaw is not a lack of validation, 


but rather the lack of either protection or interlocks that 
permitted the value to change in the first place. 

Categories 3, 5, and 6 above have been the subject of 
experimental study at ISI, including the development of 
computer tools that search given programs for specific types 
of violations—notably interprocedure data dependencies, 
potential inconsistencies of data values over time, and po¬ 
tential residues. (See Bisbey et al., 16,17 Carlstedt et a!., 18 
Carlstedt, 19 and Hollingworth and Bisbey. 20 ) However, the 
analysis of most categories is not generally amenable to 
systematic investigation. 

It is useful here to identify some of the symptoms that 
may underly flaws of each particular category. Such an 
identification is attempted in Table II, in which are listed 
various conditions that can (but not always do) cause flaws 
in security. 

DEFENSIVE SYSTEMS 

In general, good design of the hardware and software and 
good implementation are highly important. In existing sys- 

TABLE II.—Symptoms of Potential Protection Flaws, by Category 

1. Domain choice. All programs or human actions relating to the initialization 
or interpretation of protection information are suspect, e.g., any setting or 
changing of a security level, particularly any action that lessens security 
(e.g., downgrading). 

2. Exposed representations. Any direct visibility or use of implementation 
detail is suspect. Any use of absolute addresses for memory or input- 
output. Nonvirtual resources. Direct access to a data structure that is 
normally used as an abstract data object. Serial dependence within logi¬ 
cally combinational functions. 

3. Data inconsistency. A called procedure fetches the value of a parameter 
more than once, or fetches a value it just stored. A parameter is passed 
by name or by reference. (The value may change between call and 
return.) An output value is overlaid on top of an input value. Reference 
is made to a value that is self-modifying upon being accessed. (See Bisbey 
et al., 16 Carlstedt et al. 18 ) 

4. Naming. Any object for which two different names can exist is suspect. 
Use of a local name in one context, a global name in another, e.g., a 
virtual and a nonvirtual name. Any use of a table index where protection 
is expected. 

5. Residues. Physical deletion of contents is suspect whenever deferred be¬ 
yond logical deletion, e.g., deferred until reuse of media space. Readable 
free pools. Accessible backup storage. Reuse of an index or slot number 
after deletion of entry. (See Hollingworth and Bisbey, 20 which enables 
isolation of all allocations and deallocations, and searches for potential 
residues.) 

6. Nonvalidation. The absence of any checks on protection information upon 
access to sharable data is usually indicative of a flaw. The absence of any 
checks on an input variable or parameter, on its type or value range, or 
even on the existence of data is suspect. Lack of quotas on real resources 
or on different virtual partitions of resource usage can provide a leakage 
channel across partitions. Validation of status at the time of a request for 
which status may change, without revalidation on completion. (See Carl¬ 
stedt, 19 Bisbey et al. 17 .) 

7. Indivisibility. Any allegedly noninterruptible or indivisible operation is 
suspect, as is the mechanism for achieving indivisibility. 

S. Serialization. Any overlapping of operations using the same data is sus¬ 
pect, either with different uses of the same operation, or simultaneous 
uses of different operations on the same data base. 

9. Choice of operation or operand. This category is very hard to formalize. 
Potentially any operation can be improperly chosen. Operations and data 
types that do not correspond are suspect, as is the use of mismatched type 
declarations. 
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terns, it is usually bad design and bad implementation that 
lead to violations of system defenses. Thus, good design and 
good implementation should themselves be considered as 
(meta-)safeguards relating to software. The a priori use of 
good design practices and good implementation practices is 
in general enormously more effective than the a posteriori 
retrofitting of patches that attempt to strengthen the defen¬ 
ses. 

Factors influencing development of defensive systems 

The factors summarized in Table III are relevant to im¬ 
proving the defensiveness of a design and its implementa¬ 
tion. Many of these factors have profound effects throughout 
the development process, involving requirements specifi¬ 
cations, design, implementation, system maintenance, long¬ 
term evolution, and operation. 


Use of a formal development methodology for secure 

system development 

As an indication of how systems can fare with respect to 
the presence or absence of potential security flaws, a system 
is considered here that has been conceived under rather 
unusual—and hopefully optimal—conditions. This system is 
PSOS, a Provably Secure Operating System. 8 PSOS has 
been designed from the outset to be able to support ad¬ 
vanced security requirements beyond those of existing sys¬ 
tems, and has taken advantage of essentially all of the above 
factors influencing good design and implementation. In par¬ 
ticular, it has been designed according to formally stated 
requirements, has been formally specified, and has been 
subjected to formal proofs of its critical design properties. 
It is designed according to the SRI Hierarchical Design 
Methodology (HDM), described in Robinson et al., 4 Neu¬ 
mann et al., 8 and Robinson and Levitt. 21 The design is rep¬ 
resented in a formal language, SPECIAL (A SPECIfication 
and Assertion Language, Roubine and Robinson. 22 ). The 
system has not yet been implemented, but the nature of 
would-be implementations has been characterized. The de¬ 
sign has been oriented toward provability of the design and 
of subsequent implementations. (HDM is also being used in 
the design and implementation of the SIFT computer system 
for an ultrareliable commercial aircraft application [for 
which proofs of elementary fault-tolerance properties have 
been undertaken], the design of a real-time operating sys¬ 
tem, the design of a family of related message processing 
systems, and the development of provable security kernels.) 

PSOS is a capability-based system in which each capabil¬ 
ity acts as a protected name or token for an object. A 
capability for an object is protected, in that its creation and 
its transfer to other users or processes can be controlled, 
and once created, it can never be altered. It protects the 
object to which it refers, in that its presentation is required 
for that object to be accessed, including access rights ap¬ 
propriate to the operation being performed. (Certain PSOS 
applications take advantage of store-limited capabilities, 
which cannot be transferred out of the containing process.) 


TABLE III.—Factors Influencing Defensiveness in Systems and 
Applications 

Well-defined and well-understood requirements, established clearly and 
agreed upon in advance 

Good design (e.g., modularly structured, especially hierarchically, with strict 
isolation of application programs and system programs, strongly typed 
operations, unified treatment of storage, input-output [e.g., mapped virtual 
access]) 

Suitable implementation languages (e.g., strong typing, avoidance of aliasing, 
constrained argument passing [such as use of call by value where data 
inconsistency may be a problem], hiding of implementation detail and 
device dependence wherever possible, clean control structures, encapsu¬ 
lation of data types) 

Well-defined and understandable specifications for the system hardware and 
software 

Structured implementation, reflecting the modularity of the design wherever 
appropriate, and structured initialization (e.g., hierarchical) 

Systematic handling of exception conditions and quota limits 

Auditing and recovery integrated into system design, e.g., hierarchical 

Careful debugging, testing, verification 

Good management of system development (e.g., respecting these factors) 

Lessening the need for management as a result of simplifications resulting 
from use of these factors 

Good management of system operation (e.g., rigid adherence to system gen¬ 
eration and evolution protocols) 

Nonreliance on secrecy of design and implementation 

Awareness of the user community (e.g., enforcing the use of random pro¬ 
nounceable passwords rather than guessable ones) 


If formal verification of the design or its implementation is 
desired, then the following also contribute, both separately 
and collectively: 


Formally stated requirements 

Formally specified design, including specifications of modules and their in¬ 
terrelationships (e.g., data representations) 

Formal proofs of correspondence between design specifications and require¬ 
ments 

Formal axiomatization of the programming language 

Formal proofs of consistency of programs with design specifications 

Formal axiomatization of the hardware/microcode 

Formal proofs of consistency of hardware/microcode with hardware specifi¬ 
cations 


An entity analogous to a capability exists in SPECIAL, 
called a designator. A designator serves as a protected name 
of an object. Its uniqueness in SPECIAL is part of the 
specification language. In the implementation of PSOS, this 
uniqueness can be easily guaranteed by the use of capabil¬ 
ities. (Other solutions exist in other systems, such as de¬ 
scriptors.) 

Table IV gives an indication of how the use of the meth¬ 
odology together with the use of a suitable programming 
language can overcome each of the nine categories of flaws 
summarized in Table II. It is seen that the use of SPECIAL 
has a very significant impact on the avoidance of these flaws 
in design. Most of the comments on the design specification 
in the table are generally applicable to any system specified 
in SPECIAL. Similarly, the use of the hierarchical design 
methodology has significant impact on the avoidance of 
these flaws in implementation. 

The use of a suitable modern programming language can 
also contribute considerably. A language such as Euclid, 23 
Modula, 24 Texas’ Gypsy, 7 or SRI’s HDM-compatible ILPL 
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TABLE IV.—The Influence of the SRI Methodology and Programming Languages (PL) on Avoiding Characteristic Flaws, both in General and in PSOS (P:) 



Category of Flaw 

Design (in SPECIAL) 

Implementation 

1 . 

Domain choice 

Design proofs. 

Program proofs. 

2. 

Exposed representations 

Hidden by hierarchical levels of abstraction. 

PL encapsulation; proofs. 



P: Extended types further aid hiding. 

P: Extended type mechanism. 

3. 

Data inconsistency 

None. No overwrite. Conceptual call by value only. 

PL call by value; proofs. 

P: Argument values on stack unalterable. 

4. 

Naming 

Unique designators, nonbypassable, only means of nam¬ 

PL no aliasing; proofs. 



ing. 

P: Uniqueness and nonbypassability of capabilities aided 



P: Capabilities unique. 

by tagging. 

5. 

Residues 

None in specifications. Deallocated values become UN¬ 

PL dynamic object deletion and creation; proofs. 



DEFINED, are thereafter unnamable. 

P: nonreusable identifiers; contents zeroed if desirable on 



P: Residues protected. 

deallocation. 

6. 

Argument and critical condition 

Avoidable. 

PL strong typing; compile or run-time checking. 


nonvalidation 

Strong type checking, explicit subtypes, explicit exception 

P: Extended types; 1-0 addresses mapped (via capabili¬ 



and error conditions. 

ties). 

7. 

Indivisibility 

Specifications for each function are logically indivisible. 

PL synch primitives; proofs. Completion or no-effect ex¬ 
ception provable. 

8. 

Serialization 

Sequencing invisible in specifications. 

Benignness of overlap provable. 

9. 

Operation choice 

Faulty specifications detected by spec proofs. Type 
checking also helps. 

Program proofs. 


Many of the characteristic flaws can be avoided in design by the use of a methodology such as the SRI methodology, and in implementation by the use of a 
suitable programming language. 

Note: “PL” refers to language features found in such languages as Euclid, ILPL, and the emerging DoD/1 languages. “P:” refers to PSOS. (See text.) 


(due to Larry Robinson, and documented in Neumann et 
al. 8 ) would be particularly helpful, as might the languages 
emerging from the DoD/1 effort. (ILPL is an extremely basic 
intermediate-level programming language that gains its sim¬ 
plicity from the power of the supporting methodology, 
whose formal specifications provide [for example] its data 
structures as a by-product of the hierarchical levels of ab¬ 
straction.) Some of the major contributions that such lan¬ 
guages can make are indicated generically by the “PL” 
entries in Table IV. In most cases these contributions result 
from intrinsic properties of the language. In other cases they 
may result from simply enforceable language restrictions. 
Note that not all of the above languages have all of the 
desired facilities. (At present, Euclid and ILPL [the latter 
taken together with HDM] are probably the most complete 
in this respect. Modula and Gypsy are currently being ex¬ 
tended, and support for Gypsy and ILPL is currently being 
developed.) 

Finally, for many of these flaws, relatively simple proofs 
are possible to demonstrate that the particular flaw is absent 
in various explicit forms. The design aspects result from 
properties of the specifications, e.g., they follow from proofs 
of consistency between the specifications and formal re¬ 
quirements, or are intrinsic to the methodology and the 
specification language. The implementation aspects result 
from proofs of program correctness, or are intrinsic to the 
use of the programming language. With this methodology, 
it is sufficient to show consistency between the programs 
and their specifications. 21 However, many proofs are pos¬ 
sible without having to prove program correctness in gen¬ 
eral. Nevertheless, program proofs are becoming feasible as 
tools to support them continue to be developed. 

The comments in Table IV are generically applicable to 
systems designed and implemented according to the meth¬ 
odology. In addition, those prefixed with a “P:” are parti¬ 


cularized to the design and the would-be implementation of 
PSOS (e.g., in one of the above-mentioned languages). It is 
seen that a system designed according to a formal method¬ 
ology like HDM is likely to avoid the characteristic flaws 
without much additional effort. In particular, PSOS has none 
of these flaws intrinsic to its design, and it is believed that 
an implementation essentially free of these flaws can be 
developed—and if desired, proved. 

Above and beyond merely avoiding the nine characteristic 
problems, the use of HDM has a significant impact on the 
integrity of PSOS in other ways. These include the explicit 
statement of formal requirements for security, formal spec¬ 
ifications for the design and its hierarchical structure, a 
unified design, a systematic approach to the handling of 
exception conditions and quota limits, and a structured im¬ 
plementation capable of taking advantage of the hierarchical 
design without losing much in efficiency. These factors con¬ 
tribute to the suitability of the design and implementation, 
and to the possibility of carrying out formal proofs, if de¬ 
sired. This additional impact is summarized in Table V. 

Application of this evaluative approach to other systems 

The evaluation of PSOS with respect to security flaws is 
clearly favorable. This is not surprising, in that PSOS was 
designed with security in mind from the outset, using the 
formal SRI methodology (which itself evolved simultane¬ 
ously as a result of its application to PSOS). However, it is 
instructive to consider two other types of systems: the first 
which designed with security in mind, but without a formal 
methodology; the second designed with security only su¬ 
perficially in mind for its implementation. The examples 
taken are Multics and UNIX. (Conventional commercial 
operating systems are ignored here, as they are for the most 
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TABLE V.—Additional Influences of the SRI Methodology on the Integrity 
of PSOS 

Formal requirements of security: basic principles concerning acquisition and 
modification of information, and a formal model for multilevel (military) 
security. These provide simple intuitively understandable design objec¬ 
tives. 

Formal specifications of the design. These provide precise specifications for 
the system. Their consistency with the formal requirements can be formally 
proved. They provide the basis for implementation, as well as the basis for 
proofs of consistency of the implementation and the design. Proven spec¬ 
ifications also provide a strong basis for compatible alternate implementa¬ 
tions on the same or different hardware. 

Hierarchical structure of the design. This encourages separation of policy 
and mechanism, and simplifies the initial implementation significantly. Re¬ 
covery is done hierarchically, as is initialization. Recovery is therefore 
predictable and controllable. This increases the persistence of security 
during failure (partial or total). 

Unified design. Auditing functions conform to the security requirements. 
Recovery functions are also integrated into the design and use standard 
functions. Hardware design is also integrated with software design. This 
increases the ability of the hardware to provide an appropriate basis for 
the software. [Use of a pronounceable password generator might be en¬ 
forced to avoid guessable passwords.] 

Systematic handling of exception conditions and quota limits. This increases 
the predictability of system behavior, and permits proofs of completion. 
Most information channels resulting from missing exception conditions in 
specifications can be detected and eliminated. 

Implementation. To simplify the maintenance of security during system ev¬ 
olution, a highly compartmentalized implementation is very helpful. The 
hierarchical structure of the design can be retained in implementation wher¬ 
ever it is efficient to do so, and can otherwise be simplified without com¬ 
promising the security of the system. The choice of implementation lan¬ 
guage would be constrained to support data abstraction, strong typing, 
exception conditions, etc. See Table IV. 

Verifiability. Having followed the design methodology of using formal spec¬ 
ifications for the user-visible functions, it is relatively simple to prove 
properties of the PSOS design or of an application environment, e.g., 
supporting multilevel security. Similarly, with the hierarchical design and 
formal specs for internal functions, program proving seems feasible, assum¬ 
ing an appropriate choice of programming language. The language features 
noted above all tend to improve verifiability. 


part intrinsically insecure.) In essence, Multics is a large- 
scale general-purpose system with security superior to other 
commercial systems, with some emphasis on its being able 
to recover from outage, and some emphasis on audit. UNIX, 
on the other hand, makes little pretense of being secure, 
except for its attempts to provide access control and its 
encryption of the password file. It is fundamentally insecure. 

Multics, 25 like PSOS, is a system that was innovatively 
designed (albeit in 1965) to take advantage of what was then 
known about advanced design and implementation tech¬ 
niques. UNIX 26 is a system that was developed at Bell Labs 
to take advantage of then recent operating system experi¬ 
ence (e.g., Multics), on a much smaller scale—with judicious 
choice of simplifications intended to give high efficiency. 
However, UNIX was designed to be a system operating in 
a cooperative user environment, and was not conceived as 
a system providing any rugged sense of security. Neverthe¬ 
less, it is a widely used and highly useful system. The dis¬ 
cussion here is not intended in any way as a condemnation 
of UNIX as an insecure system, but rather as a simple 
illustration of how insecure a system can be in the absence 
of a pervasive concern for security from the outset. 

Table VI presents a summary of how Multics and UNIX 
fare with respect to the nine characteristic problems. It is 
clear from the table that Multics is relatively secure. It is 
equally clear that UNIX is not. Table VII presents a com¬ 
parison in terms of the methodological concepts of Table V. 

After considerable improvement upon the implementation 
of a basically sound design, Multics has reached a level of 
significant security, and has become difficult to penetrate. 
Although it was subjected to various penetration efforts on 
its earlier hardware, and was indeed penetrated, the current 
hardware and software have long since removed the flaws 
permitting those penetrations. 

UNIX has thus far apparently been used only in benign 


Category of Flaw 

1. Domain choice 

2. Exposed representation 

3. Data inconsistency 

4. Naming 

5. Residues 

6. Nonvalidation 

7. Indivisibility 

8. Serialization 

9. Op choice 


TABLE VI.—Evaluation of Multics and UNIX with Respect to Characteristic Flaws 
Multics 


Good. System, users in multiple rings. Ring 0 capture 
omnipotent, but unlikely. Outward migration of less crit¬ 
ical functions. Provable layered security kernel designed 
by Honeywell, but not implemented. 

Access within a ring all or nothing. Considerable hiding 
via ring mechanism. 

Argument pointer is copied onto stack by call. Arguments 
are copied within the system code to avoid this flaw in¬ 
ternally. 

Collisions of local names: Trojan horse! 

Core zeroed only before reallocation, but is not virtually 
addressable after deallocation. Disk never zeroed, just 
overwritten. Residues after crash. 

Hardware checking at ring and segment levels. All kernel 
args validated. Compile-time data-type checking. 

Locking makes kernel ops atomic. Exception handlers 
force no-effect noncompletion. 

Locks prevent overlap. Lock hierarchy used to avoid 
deadlocks. 

Possible problems? 


UNIX 

Poor. Nonstratified. Capture of the entire system easy 
through superuser infiltration. Security kernel implemen¬ 
tations exist (UCLA, MITRE), improving on original de¬ 
composition. 

Superuser easy to capture, defaults open. Trojan horses 
galore. Process temporaries readable and writable, main 
memory readable. 

Argument itself is copied, but only by convention. Proc¬ 
ess-temporary files are alterable in midcomputation by 
user or other users. 

Default on directory entries: unprotected. Memory aliases 
(“/dev/mem”). Trojan horses abound. 

Core zeroed only before reallocation, and is until then still 
readable. Disk never zeroed, just overwritten. Residues 
after crash. 

Very few checks on resource or 1-0 bounds, interuser 
ops. Easy to crash. 

Exception handling bad. 

OK? System functions logically synchronous (until IPC 
installed). 

Possible problems? 
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TABLE VII.—Evaluation of Multics and UNIX with Respect to 
Methodological Considerations 


Influence 

Multics 

UNIX 

Requirements 

Informal 

Informal 

Specifications 

Implementation laden 

Implementation laden 

Design structure 

Implicit hierarchy 

Procedural structure 

Design 

Fairly unified 

Fairly clean 

Except ions,quotas 

Fairly systematic 

Poor 

Implementation 

Segmentation useful 

Device independence OK 

Programming 

PL/1 somewhat more 

C poor on abstraction, 

language 

helpful than C 

strong typing 

Verifiability 

Difficult because of size, 
PL/I, no specs 

Pointless because of 
insecurity, C, no specs 


environments, and is at present simple to penetrate. It ap¬ 
pears that major renovation would be required to substan¬ 
tially improve its security (although a planned new release 
of UNIX is expected to eliminate a few of the problems). 

From the methodological considerations of Table VII, 
Multics again appears to be better off than UNIX. However, 
from this vantage point, UNIX does not seem to be irrepar¬ 
ably insecure. For example, a reimplementation with the 
imposition of constraints on C (or modifications to the lan¬ 
guage) and better handling of exceptions would be benefi¬ 
cial. 

It is interesting to note that each of these three systems 
has been considered as a basis for supporting multilevel 
security. The PSOS design is augmented with a multilevel 
security policy manager that can be efficiently implemented 
on top of PSOS (or within it, if that is to be the only policy). 14 
Multics and UNIX both have experimental redesigns ret¬ 
rofitting a kernel responsible for system security into the 
existing design. MIT, Honeywell, and the Air Force have 
undertaken the restructuring of the existing Multics system 
to improve (among other things) its security. As a compo¬ 
nent of that work, Honeywell has designed a Multics secu¬ 
rity kernel, for which formal requirements and formal spec¬ 
ifications exist (using the SRI methodology). However, the 
funding for that project (Guardian) has expired before the 
kernel could be implemented. UCLA 27 and MITRE have 
independently designed and implemented prototype se¬ 
curity kernels to be retrofitted into UNIX. These prototypes 
have provided the impetus for a competitive design effort 
between two groups (FORD-Aerospace and SRI as one 
group, TRW as the other), expected to result in a demonstra¬ 
bly secure kernel upon which is built a secure version of 
UNIX. 

CONCLUSIONS 

This document attempts to take a broad view of the devel¬ 
opment of secure systems, combining a remedial approach 
with a preventive approach. In particular, it characterizes 
various common flaws that can lead to security violations. 
It then uses this characterization to evaluate a constructive 
methodological approach to computer system development, 
and shows how such a methodology can intrinsically tend 
to avoid the characteristic flaws. This combination is 


thought to be useful for analyzing existing systems and for 
developing new systems. 

Given a computer system whose design did not originally 
have security as a major goal, and which consequently may 
be flawed, it is in general extremely difficult to converge on 
a secure system by successively patching flaws as they are 
recognized. Nevertheless, the search for such flaws is de¬ 
sirable. In general, given a system that has evolved after 
extensive penetrate-and-patch efforts, or given a computer 
system that has been expressly designed to be secure, it is 
still a difficult matter to assess how secure the system really 
is. The evaluative approach presented here is felt to be 
useful. However, compatibly with this approach, recent ad¬ 
vances in formal proof seem increasingly applicable. Such 
advances indicate the feasibility of proofs of the security of 
a design, given a suitable formal representation of the design 
and formal statements of security requirements. These proofs 
may be carried out prior to and independent of any particular 
implementation of that design. In addition, proofs of selected 
critical properties are feasible. Furthermore, the technology 
for handling proofs of consistency between design specifi¬ 
cations and programs implementing those specifications is 
progressing well. Various semiautomatic computer tools for 
carrying out such proofs now exist and are being integrated. 
However, the approach outlined here is aimed at improving 
the resulting system dramatically, even if no program proofs 
are ever carried through. 

A goal that emerges here is to be able to gain increasing 
confidence in the design and implementation throughout a 
system development, and to minimize reimplementation late 
in the development process by avoiding fundamental design 
flaws early in the process. The combination of approaches 
discussed here is seen to be relevant. A highly secure system 
can be attained, with a well-conceived design, careful im¬ 
plementation, selected formal proofs (particularly proofs of 
design properties), and the use of penetration efforts. How¬ 
ever, it should be emphasized that a completely secure sys¬ 
tem is unlikely in any case, partly because of the intrinsic 
problems presented by leakage through timing channels. 

PSOS is an example of a new system design whose stated 
goals from the beginning included support for advanced se¬ 
curity requirements and provability. The experience gained 
in that and related efforts is used here to establish criteria 
that appear to be useful in helping to evaluate other efforts. 
For example, the ISI criteria and the methodological con¬ 
siderations clearly show some of the strengths of PSOS and 
Multics. It is hoped that this paper will lead to further re¬ 
finements in security evaluation, and that it will be useful in 
evaluating other systems. It is also hoped that it will influ¬ 
ence the subsequent development of secure computer sys¬ 
tems. 
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History of programming languages 


by JEAN E. SAMMET 

IBM, Federal Systems Division 
Cambridge, Massachusetts 


This panel session will provide information on the planning, 
problems and results of a conference sponsored by ACM 
SIGPLAN held June 1-3, 1978, and entitled “History of 
Programming Languages”. This conference is intended (1) 
to initiate the preservation of a historical record for some 
major current languages and to give impetus to others to 
continue adding to this record, and (2) to provide informa¬ 
tion from one or two key contributors to the early technical 
development of the selected languages. 

Thirteen languages were selected for presentation and dis¬ 
cussion at this conference, based on the characteristics that 
the languages (1) were created and in use by 1967, (2) are in 
use in 1977, and (3) have had considerable influence on the 
field of computing. The actual criteria for choosing the lan¬ 
guages were: usage, influence on language design, overall 
impact on the environment, novelty, and uniqueness. This 
is not a conference on the entire history of programming 
languages, nor even a conference on the entire history of 
the selected languages; it covers primarily their early de¬ 
velopments, with emphasis on the technical aspects of the 
language design. The languages selected are: ALGOL 60, 
APL, APT, BASIC, COBOL, FORTRAN, GPSS, JOSS, 
JOVIAL, LISP, PL/I, SIMULA, SNOBOL. 

The organization of this conference was unusual in several 
ways. The invited speakers were assisted in preparing high 
quality papers by giving them about 13 pages of questions 
which applied to all languages, and which it was hoped 
would be answered in many of the papers. In addition to 


that, specific questions pertaining to individual languages 
were sent to each speaker. Examples of the general ques¬ 
tions are “What languages were known to you when you 
started work on your language and what influence did they 
have?” and “What class of users was your language de¬ 
signed to help?” and “To what extent did concerns about 
compilation efficiency affect the design?” Examples of spe¬ 
cific questions are “In COBOL why is there a COMPUTE 
verb allowing formulas, as well as ADD, SUBTRACT, 
MULTIPLY and DIVIDE verbs to do arithmetic” and 
“Why wasn’t there any input/output included in ALGOL 
60?” 

The problems of having contemporary technical history 
written by people who were personally involved but are not 
trained historians are formidable. They include the difficulty 
of resisting the temptation to use only memory rather than 
written documents for some material, of avoiding insults to 
people still living and actively working, of making sure that 
proper credit is given to co-workers and in general the prob¬ 
lem of handling delicate issues involving interpersonal con¬ 
flicts. 

The preprints of the papers are being produced as the 
August 1978 issue of ACM SIGPLAN Notices. Following 
the conference, a book is planned which will include these 
papers (which may be revised in the light of experience and 
comments on the presentation) and the commentary from 
the audience since the entire conference will be audiotaped 
for future listening l?y anyone. 
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COBOL—The 1980 standard, a preview 

by GEORGE N. BAIRD, MARGARET M. COOK and ROGER J. GORG 

Department of the Navy 
Washington, DC 


INTRODUCTION 

It is quite reasonable to state that COBOL is the most widely 
used programming language in the world. Originally defined 
by the Conference on Data Systems Languages (CODA- 
SYL) in 1959, COBOL has received wide acceptance. One 
of the reasons for this is the standardization of the language 
which gives the user the anticipated ability of being able to 
transport COBOL programs from one computing system to 
another without a massive conversion effort. Other reasons 
for its acceptance are its easy to understand and English-like 
structure, its ease in being maintained, and its precise data 
definition which includes editing capabilities. 

The standardization of COBOL was first accomplished in 
1968 1 after a six year effort. The subsequent conversion 
from nonstandard COBOL to COBOL 68 resulted in lan¬ 
guage conversion programs being produced which aided 
users in converting to standard COBOL compilers that were 
not necessarily compatible with their predecessors. The ad¬ 
vantage to the user in performing this conversion was to 
obtain COBOL application programs based upon a stable 
language specification which offered some guarantee that 
compilers (vendor’s version of the COBOL language), would 
also have stability between releases. 

COBOL 68 was revised in 1974. 2,3 This resulted in areas 
of incompatibility between the two standards. Some of the 
incompatibilities could not be avoided. They were the result 
of ill-defined language specifications having been included 
in COBOL 68 which were subsequently explicitly defined in 
COBOL 74 (e.g., the COBOL 68 Random Access module 
was replaced in COBOL 74 with the Relative 1-0 and In¬ 
dexed 1-0 modules). The reason for other incompatible dif¬ 
ferences could not be as easily explained. The EXAMINE 
statement being replaced by the INSPECT statement is an 
example. The EXAMINE statement could have been en¬ 
hanced to include all of the capabilities of the new INSPECT 
statement resulting in compatible programs between the two 
standards. 

The technical committee responsible for the standardiza¬ 
tion of COBOL, American National Standards Committee 
(ANSC) X3J4, is in the process of preparing for the revision 
to COBOL 74. This paper presents a preview of what the 
revised COBOL standard might be. The earliest date that a 
new COBOL Standard could be developed and approved is 


the year 1980, so this paper will refer to the revision of 
X3.23-1974 as COBOL 80. 


THE COBOL STANDARDIZATION PROCESS 

COBOL is unique in that one organization, CODASYL, 
develops the COBOL language and another organization, 
the American National Standards Institute (ANSI), stand¬ 
ardizes the language. This arrangement is beneficial in that 
two separate groups are involved with COBOL resulting in 
more immediate exposure than is afforded when a single 
committee is responsible for both the development and the 
standardization of a programming language. The negative 
aspect of the COBOL standardization process is that the 
language specifications for the revision to the COBOL 
standard must be already included in the existing standard 
or must be contained in the CODASYL COBOL Journal of 
Development (JOD) 4 as of a given date. Thus any new fea¬ 
ture must first be approved by CODASYL before ANSI can 
include it as part of a COBOL standard. 

The ANSI technical committee responsible for the stand¬ 
ardization of COBOL, X3J4, is currently making prepara¬ 
tions to produce the revision to COBOL 74. There are two 
opinions concerning this revision. One opinion is that the 
new COBOL standard should reflect the latest CODASYL 
specifications and that compatibility with the previous 
standard is not a primary consideration. The other viewpoint 
is that compatibility between versions of the COBOL stand¬ 
ard is extremely important because of the investment in 
existing programs. There will be cases where compatibility 
may be sacrificed in order to attempt to keep up with all of 
the changes made in the CODASYL COBOL Journal of 
Development since COBOL 74, the previous standard, was 
produced. 

When ANSC X3J4 is considering making changes to the 
COBOL standard during a revision process there is a spe¬ 
cific set of selection criteria which is used in determining 
the acceptability of each change. The set of criteria is as 
follows: 

(1) The general usefulness of an element or function in 
terms of: 

a. The degree of implementation; 
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b. Acceptance by users; 

c. The degree to which a function was required. 

(2) The overall functional capability of the language, con¬ 
sidering such things as redundancy. 

(3) The state-of-the-art technology with regard to imple¬ 
menting the language feature. 

(4) The usefulness, in terms of application requirements, 
of language capabilities within each level of a module. 

(5) Compatibility with other standards. 

(6) The cost of implementing versus advantages of use. 

(7) Overall language consistency within a defined level or 
module. 

(8) Upward compatibility of levels within a module. 

One must carefully examine each of these eight criteria 
and ask how the impact of any proposed change to the 
language is measurable in terms of quantity (dollars and 
cents), or whether it is measurable only in a qualitative sense 
(relatively good or bad). Vendors have a large investment 
in existing software products which include certain language 
elements and features that are extensions to the current 
standard. Users, however, have a larger investment in the 
programs they have produced using those compilers. The 
emphasis of the revision of a language standard should be 
for compatibility with the previous standard where possible. 


AN ANALYSIS OF CODASYL COBOL SINCE 1972 

The cutoff date of the CODASYL COBOL JOD for 
COBOL 74 was January 1, 1972. That was the latest dated 
document from which language elements were candidates 
for inclusion in COBOL 74. The cutoff date for COBOL 80 
will be January 1, 1978. Roughly six years of development 
of COBOL as defined by CODASYL will have taken place 
between COBOL 74 and COBOL 80. 

Since the input to the revised ANSI COBOL standard is 
limited to what is in the CODASYL COBOL JOD, the 
majority of the potential changes are already known and can 
be analyzed in terms of their impact if they are included in 
the revised COBOL standard. From a negative standpoint, 
language elements deleted from the COBOL JOD since the 
COBOL 74 cutoff date could influence the deletion of those 
elements from COBOL 80 as well. Therefore, within the 
scope of this paper, we can discuss language elements and 
capabilities which are either new, changed or deleted. The 
latter two categories will require program modification in 
order to produce programs conforming to COBOL 80 from 
programs which currently conform to COBOL 74. 

New language elements which give additional capability 

Many of the changes made to the CODASYL COBOL 
JOD have resulted in additional language capabilities. Some 
of these changes have relaxed punctuation rules while others 
have changed the category of some required reserved words 


to optional reserved words. Other changes are merely cos¬ 
metic in nature which could make program writing easier. 
These general changes will be discussed first. 


General language changes 

Source program punctuation has been clarified and sim¬ 
plified. The use of the semicolon, comma, and space can be 
used interchangeably in a source program. In COBOL 68 
the semicolon and comma could be used interchangeably 
but only in certain places. This relaxation of the punctuation 
rules will permit the semicolon and comma to appear after 
any word or character-string contained in the source pro¬ 
gram. This should reduce syntax errors due to the misuse 
of the punctuation characters semicolon and comma. 

The Environment Division which was a required division 
in a COBOL 74 program is now optional. The Configuration 
Section within the Environment Division is also optional 
when the Environment Division is present. This change was 
made primarily because of the new interprogram commu¬ 
nication facility which is described later. However it will 
simplify the production of source programs by reducing the 
number of required lines of code which must be produced. 
The presence of an Environment Division in a called pro¬ 
gram which has no files in it is a waste of time and effort. 
If the Configuration Section is specified and the Source 
Computer and Object Computer paragraph headers are pres¬ 
ent, but the computer name has not been specified, then the 
compiler will insert the computer name in the appropriate 
place(s). 

Another change which will reduce unintentional syntax 
errors in source programs permits user-defined words to be 
the same as system names supplied by the compiler writer. 
COBOL 74 prohibited these names from being identical. For 
reasons of portability and conversion this will reduce prob¬ 
lems by requiring the compiler to accept user-defined names 
which are the same as system names. The compiler in this 
case uses the context in which the word is found to distin¬ 
guish between a system name and a user-defined word. 

The order of clauses within entries in the Environment 
and Data Divisions is very rigid in some places while other 
cases permit the clauses to appear in any order. The adopted 
changes to the SELECT statement and File Description 
entries will allow their subordinate clauses to be specified 
in any order. This will make programming easier and it is 
the clause ordering procedure which has been adopted for 
other areas in COBOL. The 1-0 Control paragraph in the 
Environment Division and the Communication Description 
entry in the Data Division now permit their subordinate 
clauses to appear in any order. Another case of inconsis¬ 
tency that has also been resolved regards words which are 
sometimes optional and other times required. The COBOL 
word “IS” is optional throughout COBOL 74 except for the 
Environment Division. This difference has been resolved by 
making the presence of the word “IS” optional in the En¬ 
vironment Division as well. 

The COBOL character set, after having consisted of 51 
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characters for 18 years, has been expanded to 59 characters. 
The new characters are the commercial “@,” question mark 
“?,” colon apostrophe ampersand “&,” percent 
number sign “#,” and exclamation point The 
characters which can be used in alphanumeric literals in¬ 
clude all of the characters in a given computer’s character 
set. There is no guarantee that the characters outside the 
COBOL character set would be available on different sys¬ 
tems. The addition of these eight characters in the COBOL 
character set means they will be available on all systems 
supporting COBOL and may preclude some reprogramming 
when moving programs and data around. 

The sequence area (positions 1-6 of the COBOL source 
statement) is restricted to containing numeric characters in 
COBOL 74. The adopted changes will allow it to contain a 
combination of any characters in the computer’s character 
set. There is no requirement that the source statements be 
in any sequence based on the sequence area and there is no 
requirement that the sequence area in any particular state¬ 
ment be unique within the source program. COBOL 74 spec¬ 
ifies that nonnumeric literals cannot be longer than 120 
characters. This restriction has been lifted in CODASYL 
COBOL with no upper limit. Presumably if the standard 
reflects this change it will contain a minimum length that 
must be supported. 

An enhancement in the Data Division which will ease the 
coding of programs is the change which makes the word 
FILLER optional. COBOL 74 requires that data items which 
are not to be referenced directly in a COBOL program be 
present in such a way that they represent the data being 
ignored. The data area they represent uses the word 
FILLER in the place of a data name in the data description 
entry. The presence of the word FILLER in effect does 
nothing. Under the adopted changes FILLER will be an 
optional word so that data areas which must be defined, but 
will not be referenced directly, need not have either a data- 
name or the word FILLER specified in their data description 
entry. 

A new technique by which data can be referenced in 
COBOL has been defined. Reference modification permits 
the programmer to reference a portion of a data item by 
naming the data item, by specifying a starting position within 
the data item, and by associating a length with that particular 
reference. The starting position and length can be specified 
through variables. By this method a character string of one 
or more characters can be easily located within a string of 
data. 

COBOL 74 had a restriction that tables could only be 
three dimensions (i.e., arrays nested three deep). This re¬ 
striction has been removed and the number of dimensions 
a table may have has no upper limit. The standard, if this 
change is adopted, will probably place an upper limit on the 
number of dimensions that a table may have. 

One last inconsistency has been cleared up. This is in the 
area of character string continuation. All character strings 
can be continued in COBOL 74 except for the PICTURE 
character string. This inconsistency must have been brought 
about by an oversight during the maintenance of the lan¬ 


guage. Now at long last, a PICTURE character string can 
be continued. 

The majority of the general changes merely makes the 
rules for the language consistent throughout COBOL. Other 
changes are for the programmer’s convenience. The only 
changes which will really give additional capability are ref¬ 
erence modification and tables with dimensions greater than 
three. 


Nucleus module additions/enhancements 

The Nucleus module is the functional processing module 
of the COBOL standard. A new capability in the JOD is the 
Boolean or Bit data item. The Bit data item allows the 
programmer to define a data item in terms of bits. It can 
represent one or more bits and is used to represent data 
which can have only two distinct values, 1 and 0. 

The ACCEPT statement has been expanded to include the 
DAY-OF-WEEK as a conceptual data item from which in¬ 
formation can be obtained. DAY-OF-WEEK is composed 
of a single data element that represents the day of the 
week. The values obtained from DAY-OF-WEEK are one 
through seven. One represents Monday, two represents 
Tuesday . . . seven represents Sunday. 

The DISPLAY statement has been modified to provide a 
new capability for interacting with hardware devices for 
which positioning has meaning. When the new WITH NO 
ADVANCING phrase is specified, the hardware device will 
not be reset to the next line or changed in any way following 
the transmission of the last operand specified in that DIS¬ 
PLAY statement. If the WITH NO ADVANCING phrase 
is not specified, then after the last operand has been trans¬ 
mitted to the hardware device, the positioning of the device 
(e.g., cursor) will be reset to the leftmost position of the 
next line of the device. 

The INSPECT statement has been enhanced in two major 
areas. A new format of the INSPECT statement contains 
the CONVERTING phrase. The CONVERTING phrase 
causes the INSPECT statement to work semantically like 
the TRANSFORM statement which is supported by many 
COBOL implementations but is not contained in either 
COBOL 74 or the COBOL JOD. The data item containing 
the data to be inspected is named. Two additional operands 
indicate the characters to be converted and what each char¬ 
acter should be converted to. 

INSPECT DATA-ITEM CONVERTING “AX2” TO 
“M5B” 

DATA-ITEM (before) “1A2BWHYX” 
DATA-ITEM (after) “1MBBWHY5” 

This is a simple way to convert a number of characters in 
a string of data. COBOL 74 requires a series of “old char¬ 
acter’’ BY “new-character” phases to accomplish the same 
thing. 

A second enhancement to the INSPECT statement allows 
the inspection to be initiated and terminated within the string 
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of data being inspected. This is accomplished through the 
use of BEFORE and AFTER phrases. 

INSPECT DATA-ITEM REPLACING ‘X’ BY ‘Y’ 

AFTER INITIAL ‘A’ BEFORE INITIAL Z’ 

DATA-ITEM (before) X1A2X3Z4X5 

DATA-ITEM (after) X1A2Y3Z4X5 

Replacement begins after the first ‘A’ is located and termi¬ 
nates when the first ‘Z’ is encountered. 

Note that the operation of the above example is data 
dependent. If the ‘A’ and ‘Z’ are not encountered in the 
proper sequence, different results would occur. 

The MOVE statement has been modified so that numeric 
edited data can be moved to a numeric data item and the 
editing characters/symbols will be removed. This will help 
with data interchange among other systems and other lan¬ 
guages. When the CORRESPONDING phrase is present in 
a MOVE statement, there can be more than one receiving 
group specified. COBOL 74 only permits one receiving 
group. 

The PERFORM statement has been overhauled in an at¬ 
tempt to satisfy the syntactic and semantic needs of struc¬ 
tured programming. These will be discussed under the gen¬ 
eral type of changes for structured programming. Other 
changes in the PERFORM statement parallel the changes to 
the definition of tables. COBOL 74 allows tables to be up 
to three dimensions. With the newly adopted changes no 
upper limit will be specified. The PERFORM statement in 
COBOL 74 can vary up to three indexes/identifiers, corre¬ 
sponding to the number of dimensions permitted with a 
table. The upper limit of three varying phrases has been 
removed, permitting an unspecified number of VARYING 
phrases. Presumably, there will be an upper limit in COBOL 
80. 

The SET statement, used strictly to modify indexes as¬ 
sociated with tables in COBOL 74, has been expanded to 
include the ability to change the collating sequence used by 
the Program, the Sort/Merge process, or the Code-set for 
files during the execution of the program. It also permits the 
status of external switches to be changed and the value of 
conditional variables (88 levels) to be altered. 

There are some interesting changes which have been made 
to the Nucleus module which will make programming easier 
in some respects. However, no change of any real impact 
has been accomplished. 

Input-output module—Additions/enhancements 

A technique for establishing a padding character for se¬ 
quential files is available for inclusion in COBOL 80. 
COBOL 74 does not address padding characters, the prob¬ 
lems of a short block, or a block that is not completely filled 
with valid data records. By including the PADDING CHAR¬ 
ACTER IS clause in the Select statement for a file, the 
character specified will be used to fill out that portion of the 
block that exists beyond the last physical record. Another 
change is the RECORD DELIMITER IS phrase which can 
be used to specify an implementor defined technique for 


producing/identifying variable length records. In COBOL 74 
determination of how to produce variable length records and 
subsequently recognizing them is left to an undefined imple¬ 
mentor technique. 

The ability to specify an optional file (one which may or 
may not be present at execution time) is permitted only for 
sequentially organized files in COBOL 74. This facility has 
been expanded to include files whose organization is either 
Relative or Indexed. OPEN. . .EXTEND will also be per¬ 
mitted for Relative and Indexed files. COBOL 74 permits 
the EXTEND option (which allows records to be added 
beyond the end of a file) to be specified only for sequentially 
organized files. Another enhancement to the OPEN state¬ 
ment affects the use of the 1-0 phrase. COBOL 74 permits 
the use of the 1-0 phrase only if a file already exists. This 
causes some confusion to programmers and it makes little 
or no sense as a restriction. The file could have been opened 
for output, one record written, the file closed, and then 
opened for I-O. Now the use of the 1-0 phrase can be used 
for a file that has not been previously created. 

The RERUN statement has been modified. This statement 
is used to trigger a checkpoint dump to be taken during the 
execution of a program. The file to be used can be either an 
implementor named file or a file described in the COBOL 
program. COBOL 74 requires that if the file being used is 
described by the program that it be a sequentially organized 
file. The rules also have been changed so that a relative or 
indexed file can be used as well. The wisdom of using a 
relative or indexed file for storing checkpoints may be ques¬ 
tionable but the capability now exists. 

The MULTIPLE FILE TAPE clause found in the Input- 
Output Control paragraph has been clarified. In COBOL 74 
there is a question regarding whether a File Description 
entry must be present in the program for files which are on 
the tape but are not going to be referenced. Under the 
adopted changes the introduction of a pseudo-file-name will 
be used to identify these files. The new facility also allows 
a multiple file tape to be contained on more than one reel or 
physical volume. 

Variable length records will be requested using the new 
form of the RECORD clause in the File Description Entry 
for those files. 

RECORD IS VARYING IN SIZE FROM SHORT- 

LENGTH TO LONG-LENGTH CHARACTERS 

DEPENDING ON LENGTH-INDICATOR 

The minimum and maximum record lengths are specified 
in the RECORD VARYING clause. The DEPENDING ON 
phrase names the data item which will be used to define the 
length of the current record. This data item will be defined 
in the Working Storage Of Linkage Section. Prior to the 
record being written, the length of the current record is 
placed in the Depending On data item by the programmer. 
This establishes the logical length of the record to be written. 
Later when the record is read, the input/output control sys¬ 
tem places the length of the record in the Depending On 
data item. Using this value the programmer then will know 
the size of the record just read. 
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The LABEL RECORDS clause has been made optional, 
whereas COBOL 74 requires the clause be present for each 
file described in the source program. When the LABEL 
RECORDS clause is not specified, the file will be treated as 
though the STANDARD option had been specified. Another 
change makes the words RECORD/RECORDS optional 
words in both the LABEL RECORDS and DATA REC¬ 
ORDS clauses. 

The REEL/UNIT phrase can be used in a CLOSE state¬ 
ment for a sequentially organized file which is a report file 
(i.e., produced by the COBOL Report Writer). This is ex¬ 
plicitly prohibited in COBOL 74. The usefulness of this 
function may be questioned since a report file is subse¬ 
quently passed to the operating system for printing. 

The DELETE statement has been modified so that a file 
can be referenced as well as a record within a file. When 
the FILE format of the DELETE statement is used, the 
referenced file is deleted and any associated storage medium 
is released and made available to the operating system. The 
file being deleted must not be a sequentially organized file. 
After the DELETE statement has been executed, the file is 
no longer available to the Input-Output Control System. 

Somewhat related to the Input-Output modules is the Sort- 
Merge module. Only a few additions have been made. Both 
the SORT and MERGE statements can have multiple files 
named in the GIVING phrase. This permits two or more 
copies to be created of the file produced as a result of a 
SORT or MERGE operation. In addition, the GIVING 
phrase of a SORT or MERGE statement now may reference 
relative or indexed files. COBOL 74 requires that the giving 
file be a sequentially organized file. 

The SORT statement has a new phrase, the WITH DU¬ 
PLICATES IN SEQUENCE phrase, which will cause rec¬ 
ords whose keys are identical to remain in the sequence in 
which they are input to the sort process. The absence of this 
phrase, as in COBOL 74, leaves the task of sorting records 
with duplicate key-values to the implementor. 

Inter-program communication module additions/ 
enhancements 

One of the two major enhancements to CODASYL 
COBOL since the COBOL 74 cut-off date is the modification 
made to the interprogram communication facility (CALL 
statement). The impetus to make this modification was the 
recent emphasis on software engineering and structured pro¬ 
gramming. The other major enhancement to COBOL is the 
inclusion of the syntax and semantic forms necessary to be 
able to do structured programming in COBOL directly with¬ 
out having to simulate the major structured programming 
constructs. The structured programming aspect of COBOL 
will be addressed later. 

COBOL 74 suffers in software engineering circles due to 
the structure of the language. Data within a given program 
is available throughout the program. There is no method by 
which data for a given set of procedures can be earmarked 
as being available only to those procedures. The JOD 
COBOL has been modified to provide a solution to these 
problems. 


Source programs can be contained within other source 
programs. The contained source programs are referenced by 
the CALL statement. A program will always be in its initial 
state when called if the program is identified as being an 
INITIAL program. If a contained program is to be accessible 
to programs other than the program which contains it, its 
PROGRAM-ID statement must indicate that the program is 
COMMON. The END PROGRAM statement terminates a 
contained program. 

IDENTIFICATION DIVISION 
PROGRAM-ID. PROG1. 


IDENTIFICATION DIVISION 
PROGRAM-ID. PROG2 IS COMMON INITIAL 
PROGRAM 


END PROGRAM PROG2. 


END PROGRAM PROG1. 

PROG2 is a source program contained in PROG1. When 
PROG1 calls PROG2 parameters can be passed via the 
USING phrase of the CALL statement, as was the case in 
COBOL 74. Further, data in the contained program 
(PROG2) described as EXTERNAL can be referenced by 
the containing program and by any other program in the run 
unit. The data items in the referencing programs must be 
described in the same manner and using the same data- 
names and equivalent attributes as were used to define the 
data items in the called program. Names in the containing 
and contained programs that are identical, but not identified 
as being EXTERNAL in the contained program, reference 
unique items within each program. Data items in the con¬ 
taining program (PROG1) identified as GLOBAL can be 
referenced by the contained program (PROG2). The names 
of GLOBAL data items must not be duplicated in other 
programs referencing them. 

There is also an additional option in the passing of param¬ 
eters when using the CALL statement. If the BY CON¬ 
TENT phrase is used in naming a parameter to be passed, 
then the called program can modify its version of the param¬ 
eter but it has no effect on the value of the parameter in the 
calling program. If the BY REFERENCE phrase is used in 
naming parameters, the result is the same as it is in COBOL 
74. Any changes to the data in the called program will affect 
the data in the calling program. 

Structured programming enhancement/additions 

COBOL 74 has, at best, co-existed with structured pro¬ 
gramming. The basic control structures are not available in 
COBOL 74 and have to be simulated using variations of the 
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PERFORM statement. Additionally, the number of places 
that asynchronous interrupts can occur in COBOL is well 
beyond what is suggested in structured programming. 

IF-THEN-ELSE and Asynchronous interrupts 

The problems associated with the COBOL 74 IF statement 
have been solved in the JOD COBOL. Pure nesting of the 
COBOL 74 IF statement was precluded because of the syn¬ 
tax of the language. The END-IF phrase has been introduced 
to make the IF statement more functional. 

IF 

Condition 

THEN 
statement-1 

ELSE 

statement-2 

END-IF 

In COBOL 74 an IF statement is terminated by an ELSE 
phrase associated with an IF statement at a higher level or 
by the separator period. Using the JOD COBOL the END- 
IF phrase can be used to specifically terminate a nested IF 
statement. 

The statements containing asynchronous interrupts have 
been handled in a similar fashion. 

ADD identifier TO identifier 
ON SIZE ERROR imperative-1 
NOT ON SIZE ERROR imperative-2 

END-ADD 

Using the ADD statement as an example the ON SIZE 
ERROR phrase illustrates the true line path of the condition, 
whereas, the NOT ON SIZE ERROR phrase is the ELSE 
(or false) side of the condition. The END-ADD phrase ter¬ 
minates the ADD statement and its inter-related conditional 
statement. The END-verb phrase has also been included in 
all statements which can cause an asynchronous interrupt 
(i.e., statements containing the ON SIZE ERROR, AT END, 
ON EXCEPTION, INVALID KEY, ON OVERFLOW, 
WHEN, and NO DATA phrases.). 

DO-WHILE and DO-UNTIL 

The PERFORM statement in JOD COBOL has been ex¬ 
panded to include an inline looping capability. 


By specifying the TEST BEFORE phrase, the procedural 
code will be executed as a DO-WHILE construct. If the 
TEST AFTER phrase is specified, then the procedural code 
will be executed as a DO-UNTIL construct. 

The EVALUATE statment, which is much too complex 
for the scope of this paper, represents CODASYL’s answer 
to the CASE construct in structured programming. The 
EVALUATE statement can also be used to express a de¬ 
cision table in a COBOL format. 


Incompatible changes to/deletion of language elements 

As the CODASYL COBOL specifications evolve changes 
are made to the existing syntax and semantic actions of 
language elements; and language elements are deleted. 
These changes, if reflected in COBOL 80, will require source 
programs to be modified in some cases quite extensively. 
The most drastic of the incompatible changes are itemized 
in the following paragraphs. Language elements which have 
been deleted from CODASYL COBOL are: 

• The 77 level data item. The rationale for this deletion 
is that an elementary 01 level data item accomplishes 
the same thing. 

• The ALTER statement. The value of using the ALTER 
statement has been questioned for some time due to the 
way in which it can obscure the logic of a program. 

• Numeric Paragraph and Section names in the Procedure 
Division. 

• The REVERSED phrase of the OPEN statement. 

• Initial segments (Segmentation Module) 50-99. 

The following language elements have been modified such 
that their inclusion in COBOL 80 will require source pro¬ 
grams to be changed in order to continue to use the function. 

• The BLOCK CONTAINS clause, and the CODE-SET 
clause have been removed from the File Description 
entry in the Data Division and placed in the SELECT 
statement of the Environment Division. 

• The ACCESS MODE, RECORD KEY, ALTERNATE 
RECORD KEY and FILE STATUS clauses have been 
deleted from the SELECT statements in the Environ¬ 
ment Division and placed in the File Description entry 
of the Data Division. 

• The rules for the RECORD CONTAINS clause state 
that the absence of the clause will cause fixed length 
records to be written regardless of the record descrip¬ 
tions. COBOL 74 treated the RECORD CONTAINS 
clause as a comment and used the actual record de¬ 
scriptions to determine the length of records. 

SORT or MERGE files cannot be named in a RERUN 
statement to trigger a check point to be taken during the 
execution of a Sort or Merge operation. Also, a SORT 
or MERGE statement cannot be referenced in a USE 
FOR EXCEPTION statement in the DECLARATIVE 
portion of the program. 

For the INSPECT statement (remember the EXAMINE 


PERFORM WITH TEST 


BEFORE' 
AFTER 


Procedural Code 


END-PERFORM 
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statement), both TALLYING and REPLACING can¬ 
not be included in the same statement as was permitted 
in COBOL 74. 

• The description of the DEBUG-ITEM associated with 
Debugging procedures has been changed. 

• All files included on a multiple file tape must have the 
same labeling scheme, i.e., STANDARD or OMIT¬ 
TED. COBOL 74 has no such restrictions. 

• The REDEFINES clause cannot be specified in the 
Data Description entry for an index data item. 

The changes described in this section, with the exception 
of the deletion of the ALTER statement, are questionable. 
If all of these changes are reflected in COBOL 80, every 
COBOL program ever written would have to be modified to 
be acceptable to a COBOL 80 compiler. There is a possi¬ 
bility that X3J4 will allow a transition period for some of the 
incompatible changes; for example, the clauses that have 
been moved from the File Description entry to the SELECT 
statement in CODASYL COBOL will appear in both places 
in COBOL 80. Language elements in COBOL 80 will be 
annotated to indicate that they will be deleted when COBOL 
80 is revised. This will give the COBOL user advanced 
warning about potential changes to COBOL which impact 
existing programs. 


CONCLUSION 

The revision cycle for developing COBOL 80 from COBOL 
74 and the 1978 JOD is under way. It seems appropriate at 
this point to ask two basic questions. Based on the enhance¬ 


ments which can be included in COBOL 80, is the industry 
ready for a new COBOL standard? What would COBOL 
users gain by adoption of a new standard? 

COBOL compilers based on COBOL 74 have only been 
available since late 1975 and early 1976, and some vendors 
had not released COBOL 74 compilers by the end of 1977. 
Most of the COBOL community has not yet undergone the 
conversion effort required for changing COBOL 68 pro¬ 
grams to COBOL 74. After they have gone through this 
process, most will not be ready to accept another conversion 
effort five years later. 

Once a revision cycle is started, it develops its own mo¬ 
mentum, and the question of whether the user community 
is ready for a new standard may be lost. With the exception 
of reference modification, variable length records, and struc¬ 
tured programming changes, there does not appear to be 
any enhancements justifying the program modifications 
which would be required, and the increase in size and cost 
of COBOL compilers. The COBOL copimunity has the re¬ 
sponsibility to communicate their views on these questions 
to the American National Standards Institute. 
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INTRODUCTION 

The American National Standards Committee (ANSC) X3J4 
is responsible for developing the standard for the program¬ 
ming language COBOL. ANSC X3J4 is currently in the 
process of revising American National Standard Program¬ 
ming Language COBOL, X3.23-1974. 1 As part of this revi¬ 
sion process, X3J4 is developing a candidate Database mod¬ 
ule for inclusion in the next COBOL Standard. The earliest 
date that a new COBOL Standard could be developed and 
approved is the year 1980, so this paper will refer to the 
revision of X3.23-1974 as COBOL 80. 

Direction for developing the COBOL Standard is given to 
X3J4 by its parent committee, ANSC X3, the ANSI tech¬ 
nical committee responsible for Computers and Information 
Processing. According to X3 direction, the source of the 
language specifications for COBOL 80 must be either the 
current COBOL Standard or the CODASYL COBOL Jour¬ 
nal of Development. 2 Thus the only possible candidate for 
standardization of a Data Manipulation Language (DML) for 
COBOL is the CODASYL database facility. X3J4 cannot 
consider any other database model, such as relational, for 
standardization. 

In order to develop a candidate Database module for 
COBOL 80, X3J4 has established a Database Task Group, 
X3J42, which has the responsibility of reviewing the data¬ 
base facility specifications in the CODASYL COBOL Jour¬ 
nal of Development (JOD) and identifying those elements 
which are to be included in the candidate Database module. 
The JOD cutoff date for COBOL 80 has been established as 
January 1, 1978. Database elements in the JOD as of the 
cutoff date are the source of the database specifications 
which can be included in COBOL 80. 

This paper discusses the structures which can be defined 
according to the CODASYL database specifications and the 
language elements which are available for standardization. 
The standardization effort by X3J4, and hence its subcom¬ 
mittee X3J42, is limited to that portion of a Database Man¬ 
agement System (DBMS) which pertains to the defining of 
a Sub-schema and the Data Manipulation Language state¬ 
ments proposed for the host language COBOL. The portions 
of a DBMS which are outside the purview of COBOL, such 
as the Data Definition Language and the Device Media Con¬ 
trol Language, are not candidates for standardization by 


X3J4. However, the functions and structures of the Data 
Definition Language, because of their close relationship to 
the Sub-schema definition, are discussed in this paper. 

OVERVIEW OF THE DATA BASE FACILITY 

The portions of the CODASYL database facility which 
are of concern to X3J4 are the database functions which 
permit an application program written in COBOL to com¬ 
municate with the Database Management System. The func¬ 
tions required are a definition of that portion of a data base 
of interest to an application program and the procedural 
statements required to access the database. 

The entire database is defined in the Schema. The Schema 
contains a description of inter-record relationships and the 
data items comprising each record in the database. Because 
the Sub-schema definition is a subset of the Schema defi¬ 
nition, understanding the Schema and its capabilities is a 
prerequisite for understanding the Sub-schema definition. 

The COBOL Sub-schema defines the portion of the da¬ 
tabase available to an application program. The part of the 
database that is not required by a given application area is 
not defined in the Sub-schema. The Sub-schema definition 
is a subset of the Schema definition but can define data 
items in a different manner than the Schema, as long as the 
definition is consistent with the Schema. 

The COBOL Data Manipulation Language statements for 
the database facility contain update functions for data items 
and record relationships, and record and item retrieval func¬ 
tions. The DML statements are contained in the Procedure 
Division of a COBOL program and can only reference rec¬ 
ords and record structures which have been defined in the 
Sub-schema referenced by the COBOL program. 

The Schema, COBOL Sub-schema and Procedure Divi¬ 
sion DML statements combined provide a database facility 
through the host language COBOL. The succeeding sections 
of this paper discuss each of these components in detail. 

THE SCHEMA DEFINITION 

The logical definition of the database is provided by the 
Schema. The Schema defines the record types in the data¬ 
base, the inter-record relationships, and the access control 
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mechanisms. The specifications for the Schema definition 
are contained in the CODASYL Data Definition Language 
Journal of Development. 3 


Record definition 

The basic unit of access in the CODASYL database fa¬ 
cility is the record. Each record type in the database is 
defined in the Schema. The record definition specifies the 
data items and data aggregates which comprise records of 
a particular record type. The smallest unit of data is the data 
item. In data subentries within a record definition, complete 
descriptions are given for each data item in the record. 
Among the data characteristics specified are the data type, 
that is, arithmetic, character or bit string, system generated 
identifier (database key), or implementor-defined extension; 
validation values for checking accuracy of data item con¬ 
tents when storing items in the database, and access control 
locks for retrieving, storing, or modifying data items. A 
named collection of data items in a record type is called a 
data aggregate and permits the definition of vectors and 
repeating groups. 

In addition to data items whose contents are supplied by 
the user, the definition of derived data items is also permit¬ 
ted in the CODASYL Schema. Derived data items are data 
items whose contents are generated by the DBMS according 
to specifications in the data subentry. There are two types 
of derived data items, actual and virtual. The DBMS gen¬ 
erates the contents of an actual derived data item during a 
store or modify and the contents are stored as part of the 
record. A virtual derived data item is generated by the 
DBMS when a record is retrieved from the database. The 
item does not exist in the database but is available to a user 
program as if it were stored in the database. 

Within the Schema definition an arbitrary number of rec¬ 
ord types may be defined, and there is a record description 
for each type of record in the database. For any given record 
type defined in the Schema, there may be none, one, or an 
arbitrary number of occurrences of records in the database. 


Set definition 

A Set is the basic structure by which inter-record rela¬ 
tionships are defined in a CODASYL Schema. Unfortu¬ 
nately, the use of the word set is confusing to those familiar 
with set as a mathematical entity. The misuse of the letters 
“SET” according to Steele is “That linguistic atrocity per¬ 
petrated by the DBTG Report wherein the nineteenth, fifth, 
and twentieth letters of the Roman alphabet are used in that 
order as the name of a peculiar object. This may seem harsh, 
but the point at issue represents a prize example of the 
manner in which the information processing sciences gen¬ 
erate confusion for themselves and others by casual misuse 
of words.” 4 

A set type in the Schema is defined by a single owner 


record type and one or more member record types. A record 
type may be defined to be the member of more than one set 
type, and a member of one set type may be the owner of a 
different set type. The set type relationship permits the 
definition of network structures in the CODASYL database 
facility. 

There are restrictions on the record relationships permit¬ 
ted in the Schema set definition. A record type cannot be 
defined as both owner and member of the same set type, 
and a set type may not have more than one owner record 
type. Thus the many-to-many owner-member relationship is 
not directly supported by the CODASYL structure. 

Set membership 

There are two classes of set membership defined for each 
record type. The first class concerns the insertion of records 
into a set, and the second class concerns the removal of 
records from a set. For the insertion of records the member 
record types can be either automatic or manual. If automatic 
membership is specified, membership in a set is established 
by the DBMS when a record is created in the database. If 
manual membership is specified, a Data Manipulation Lan¬ 
guage statement is required to connect an already existing 
record in the database to its owner record. 

For the removal of records from a set, the member record 
types can be either mandatory or optional. If mandatory 
membership is specified, once a record is a member of a set 
it cannot be removed from membership in that set type. If 
optional membership is specified, a member record can be 
disconnected from a set of the given set type by the exe¬ 
cution of a DML statement. 


Set selection 

The method for the selection of a specific set from the 
totality of sets in the database is specified in the set selection 
criteria. The identifiers are specified which are to be used 
in navigating from owner record to member record in a 
series of sets on a continuous path, where the member of a 
given set is the owner of the next set in the path. The values 
of the named data items are used to select a specific set in 
the database. 


Set order 

Each set type defined in the Schema must have an order 
specified for the member records of a set of the given set 
type. The member records may be ordered by ascending or 
descending keys. The keys can be data items, member rec¬ 
ord names, or system generated identifiers. The member 
records may also be ordered based upon the insertion order 
of new member records into the set, for example, first, last, 
or before another member record selected by an application 
program. 
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REALMS 

In the Schema the database can be defined to be logically 
divided into realms (areas). A realm is similar to a COBOL 
file in that a Data Manipulation Language statement is re¬ 
quired before a program can access records within a given 
realm. The record type definition specifies the realms in 
which the records of a particular type may appear. An in¬ 
dividual record can be associated with only one realm, and 
a record may not change realms. 

There may be an arbitrary number of realms defined in 
the Schema, and records can be assigned to realms inde¬ 
pendently of their inter-record relationships (set structure). 
An owner record and member records of a set may be 
contained in the same realm or in many different realms. 
Similarly, a given record type or set type may have occur¬ 
rences which appear in the same realm or in many different 
realms. 

COBOL SUB-SCHEMA DESCRIPTION 

A COBOL Sub-schema definition specifies the data items, 
record types, set types and realms which can be accessed 
by a COBOL program. A COBOL Sub-schema contains an 
individual program’s view of the database. Only those data 
items, records, sets, and realms described in a COBOL Sub¬ 
schema may be accessed in a COBOL program referencing 
that Sub-schema. The portions of a database beyond the 
scope of an individual program are not defined in the Sub¬ 
schema referenced by the COBOL program and are not 
accessible to the program. 

There may be an arbitrary number of COBOL Sub-sche¬ 
mas described for the Schema, and definitions for different 
Sub-schemas may reference the same Schema entries. A 
COBOL program references only one Sub-schema, but an 
arbitrary number of programs may reference the same Sub¬ 
schema. 

COBOL Subschema relationship to schema 

A COBOL Sub-schema definition is a subset of the da¬ 
tabase description given by the Schema. A Sub-schema 
specifies the data items, records, sets and realms which can 
be referenced by a COBOL program. A Sub-schema cannot 
define new record types or set types, and all data defined in 
the Sub-schema must have been previously described in the 
Schema. 

Important differences in the definitions for the Schema 
and COBOL Sub-schema are permitted in the CODASYL 
database facility. The Database Management System uses 
the Schema and COBOL Sub-schema definitions to perform 
the data conversion required for access by a COBOL pro¬ 
gram. 

The Sub-schema references data items and records in the 
Schema by their names given in the Schema. The names for 
identical data items in the Sub-schema and Schema can 


differ if they are the subject of an Alias Description entry, 
in which case use of the Sub-schema names is equivalent to 
use of the Schema names. The COBOL program always 
references the data-names defined in the Sub-schema. 

Only a portion of a Schema record need be defined in the 
Sub-schema record and the data items in the Sub-schema 
may be specified in a different order. The data mapping is 
performed by matching data-names and record-names to the 
data-names and record-names in the Schema. The mapping 
is performed at the elementary item level, and the data 
description in the Sub-schema is the description used by the 
COBOL program. 

The characteristics of data items defined in the Sub¬ 
schema can differ from the Schema specifications as long as 
the results are consistent with the Schema definition. 
Data transformation of equivalent data items is performed 
according to the MOVE statement rules for elementary 
sending and receiving items. Alternately, a database pro¬ 
cedure could be defined in the Schema which performs the 
required transformation, in which case the MOVE statement 
rules do not apply. 

Data item validation is performed by the DBMS as it 
stores data into the database or retrieves data from the 
database. The data item validation values specified in the 
Sub-schema may be different from the validation values 
specified in the Schema, but the Sub-schema specification 
must be within the range of the Schema specification. The 
DBMS checks the contents of data items when retrieving 
them from the database and does not retrieve items outside 
of the Sub-schema specified range. 

Access control locks for a given realm, set type, record 
type, or data item can be declared in either the Schema or 
Sub-schema, or in both. If an access control lock for a given 
entity has been declared in both the Schema and Sub¬ 
schema, the Sub-schema access control lock overrides the 
Schema lock and the Sub-schema access control lock must 
be satisfied by the COBOL program. 

The set description in the Sub-schema permits the speci¬ 
fication of the arguments to be used by the DBMS in defining 
the set selection criteria. If set selection criteria are defined 
for a record type in the Sub-schema, they replace the set 
selection criteria defined in the Schema. If set selection 
criteria for a record type are not defined in the Sub-schema, 
then the Schema set selection criteria remain in effect. 

Data independence 

Although the data items defined in the Sub-schema must 
be defined in the Schema, important differences are permit¬ 
ted which give COBOL programs data independence. The 
definition of data in the database is contained in the Schema 
which is a step removed from the COBOL program’s view 
of the database as given in the Sub-schema. Because of the 
differences between the Schema and Sub-schema which are 
permitted in the CODASYL database facility, changes can 
be made to the Schema without making any modifications 
to the programs accessing the database. 
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Modification of the Schema may require modification of 
the Sub-schemas based on the Schema. If the changes re¬ 
define the set structures or remove data items from a record 
type, then changes to the Sub-schemas referencing these 
structures and items must be made. If the changes are the 
addition of new data items, record types, and set structures 
which are not referenced by the Sub-schema, or if the data 
descriptions are changed, Sub-schema modifications are not 
required. 


COBOL Subschema divisions 

A COBOL Sub-schema source program consists of three 
divisions: Title, Mapping, and Structure. The Title Division 
specifies the name of the Sub-schema and the name of the 
Schema to which the Sub-schema is related, the Sub-schema 
access control key which must satisfy the Schema access 
control lock in order for the Sub-schema to be placed in the 
Sub-schema library, and the access control locks associated 
with the use of the Sub-schema. The Mapping Division con¬ 
tains the Alias Section where Sub-schema names not defined 
in the Schema are equated to names defined in the Schema. 
Identifiers, realm-names, record-names, and set-names may 
be named in the Alias Section. 

The Structure Division contains the Realm Section, the 
Set Section, and the Record Section. This division specifies 
the portion of the database that is to be made available to 
a COBOL program and the structure of that portion of the 
database. The Realm Section specifies the Sub-schema 
realms and the access control locks associated with the use 
of the realms. The Set Section specifies the Schema set 
types to be made available to the Sub-schema, the set se¬ 
lection criteria, and access control locks associated with the 
use of the Sub-schema sets. The Record Section specifies 
the Schema record types and the data items within the rec¬ 
ord types that are to be made available to a Sub-schema. 
Access control locks associated with the use of records and 
data items, and the data item characteristics are also spec¬ 
ified in the Record Section. 


Compilation and binding time of COBOL Subschema 

The Schema, COBOL Sub-schema and COBOL program 
are required to operate as a unit in order to perform database 
functions. The time of compilation of the Sub-schema and 
the binding time of the Schema, Sub-schema, and COBOL 
program as a run unit are extremely important. Some of the 
possible times for binding are COBOL compile time, 
COBOL Sub-schema compile time, execution time, or any 
other implementor-defined time. 

The results for a program using the database facility could 
vary depending upon the binding time. For example, if a 
Sub-schema and COBOL program are compiled and bound 
together with a Schema as a run unit, and the Schema 
definition is then changed, the results of the execution could 
be quite different from compiling and binding the Sub¬ 


schema and COBOL program with the Schema after the 
changes are made to the Schema. 

COBOL 80 Subschema subsetting 

If a COBOL database facility is included in COBOL 80, 
it is quite probable that the COBOL 80 Sub-schema will be 
a subset of the COBOL Sub-schema specifications in the 
CODASYL COBOL Journal of Development. The X3/ 
SPARC Database Systems Study Group has expressed con¬ 
cern over declarations in the Schema being overridden by 
declarations in the Sub-schema. Votes taken by ANSC X3J4 
indicate that this concern is shared by X3J4. 5 It is probable 
that the specifications for access control locks for realms, 
set types, record types and data items will not be contained 
in the COBOL 80 Sub-schema. If access control locks were 
permitted for these resources, they would take precedence 
over access control locks specified in the Schema for the 
same resources. 

The set selection criteria specified in the Sub-schema re¬ 
places the set selection criteria in the Schema and may not 
be included in the COBOL 80 Sub-schema for that reason. 
In the case of the set selection criteria, there is sentiment in 
X3J42 that the Sub-schema override of the Schema is not a 
problem and set selection criteria should be included in the 
COBOL 80 Sub-schema. 

DATA DIVISION SUB-SCHEMA SECTION 

If the database facility is included in COBOL 80, the Sub¬ 
schema Section will be added to the Data Division. The Sub¬ 
schema Section consists of a Sub-schema entry which spec¬ 
ifies the COBOL Sub-schema that is to be accessed by the 
COBOL program, and the access control key which must 
satisfy the access control lock associated with the Sub¬ 
schema. 

Only one Sub-schema entry is permitted per COBOL pro¬ 
gram. Multiple Sub-schemas per program are not presently 
permitted by the CODASYL Journal of Development and 
hence cannot be permitted by COBOL 80. 

PROCEDURE DIVISION DML STATEMENTS 

The Procedure Division Data Manipulation Language 
statements provide an interface to the database for a 
COBOL program. The Procedure Division DML statements 
provide the capability to update the database, to retrieve 
records and data items from the database, and to control the 
availability of realms to the COBOL program. 

Update function statements 

There are DML statements which provide update func¬ 
tions for both set structures, records, and data items within 
a record. Records are stored in the database by executing 
a STORE statement. The record stored becomes a member 
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of each set type in which it has been declared in the Schema 
to be an automatic member, and the record becomes the 
owner of an empty set (set with no members) for each set 
type for which it has been defined to be an owner. Records 
are removed from the database by executing an ERASE 
statement. The ERASE statement can remove one record, 
or with the use of additional options, all records within a set 
structure for which the current record is the owner record. 

A record becomes a member in one or more sets for which 
the record type has been defined to be a manual member by 
executing a CONNECT statement. A record is removed 
from one or more sets for which its record type has been 
defined to be optional by executing a DISCONNECT state¬ 
ment. 

The MODIFY statement is used to change the contents 
of one or more data items in a record, the set membership 
of a record, or to change both data items within a record 
and the set membership of a record. Execution of a 
MODIFY statement without the ONLY option causes the 
DBMS to replace in the database the contents of data items 
within the current record with the contents of the record 
area associated with the record. Derived data items may be 
indirectly changed by execution of a MODIFY statement. 
If the INCLUDING option or the ONLY option is specified 
in a MODIFY statement, the set membership of the current 
record is changed in accordance with the set ordering criteria 
of each set type. 

The ORDER statement permits the logical reordering of 
members of the set of which the current record is either an 
owner or a member record. The logical reordering may apply 
to only the current run unit, or the reordering of the set 
member records may occur in the database. The object 
records of an ORDER statement may be all the records 
referenced by a given record-name, all records in which a 
given data-name is defined, or all records that are members 
of the set containing the current record. 

Retrieval function statements 

The two DML statements which are used to locate records 
and retrieve all or part of a located record are the FIND and 
GET statements. The FIND statement is used to locate a 
specific record in the database which then becomes the 
current record of the run unit. Execution of the FIND state¬ 
ment does not retrieve the selected record but identifies it 
as the current record for use in subsequent DML statements. 
The record selection expression in the FIND statement 
specifies the criteria to be used in determining the object 
record of a FIND statement. The record selection criteria 
may be based on record contents, system generated identi¬ 
fiers, or a record’s relationship with other records. 

The GET statement is used to move all or part of a record 
from the database into its associated record area where it 
can be accessed by other Procedure Division statements in 
a COBOL program. The GET statement without any iden¬ 
tifier specified or with identifier consisting of a record-name, 
moves the portion of a database record defined in the Sub¬ 
schema into its associated record area. If identifiers refer¬ 


encing data items are specified, the contents of those data 
items are moved into the associated record area. Data items 
not referenced are unchanged in the record area. In all cases 
the current record of the run unit is the database record 
accessed. 


Control function statements 

There are two DML statements which control a COBOL 
program’s access to database realms. The READY state¬ 
ment prepares one or more realms for processing and spec¬ 
ifies whether the records in a realm can be accessed and 
modified, or only accessed. The READY statement also 
indicates whether the records in a realm are for the exclusive 
use of the run unit, or if other run units are allowed to 
access the realm but prevented from updating the specified 
realm. 

The FINISH statement terminates the availability of one 
or more realms to the run unit. The individual realms to be 
terminated can be specified by name, or a set name can be 
referenced in which case all realms which contain the owner 
or members of the named set are terminated. 

The DML includes the COBOL IF statement with data¬ 
base conditions as the conditional expression to be tested. 
There are three database conditions which may be tested: 
Tenancy, Member, and Nullity. The Tenancy condition de¬ 
termines if the correct record of the run unit is the owner, 
member, or either an owner or a member of a set of a given 
set type. The Member condition determines if a given set 
has any member records. The Nullity condition is used to 
determine if a specified data item has the null attribute. 

USE declarative statement 

The USE declarative statement for the COBOL data base 
facility specifies the procedures to be used in producing 
access control keys for satisfying Schema and Sub-schema 
access control locks pertaining to functions performed on 
realms, records, sets or identifiers. The USE statement also 
specifies the procedures to be executed on database excep¬ 
tion conditions. 

Concurrent run units 

There are three DML statements which are included in 
the COBOL database facility to monitor concurrent usage 
of records by different run units. The three COBOL state¬ 
ments are KEEP, FREE, AND REMONITOR. The entire 
area of monitored mode and the functions of these verbs are 
expected to change from those specified in the Journal of 
Development in the Fall of 1977, so these statements will 
not be further explained. ANSC X3J4 has agreed that the 
KEEP, FREE, and REMONITOR statements, and the ex¬ 
tended monitored mode as they are specified in the Fall 1977 
COBOL JOD are not suitable for standardization and hence 
will be excluded from COBOL 80. 
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COBOL 80 DML statements subsetting 

As mentioned earlier in the Sub-schema explanation, 
ANSC X3J4 is concerned about Sub-schema declarations 
replacing Schema declarations, and Sub-schema declara¬ 
tions being overridden by programmer action. Votes taken 
by X3J4 give an indication of the possible subsetting which 
will occur in determining the DML functions suitable for 
standardization. 5 * 6 The ORDER, KEEP, FREE, and RE¬ 
MONITOR statements have been identified as statements 
which will not be included in COBOL 80. Also, it is probable 
that the USE statement for data base functions will be subset 
for COBOL 80. 

There are many problem areas which CODASYL is trying 
to resolve before the cutoff date. Among the data base 
features which have been identified as problem areas are the 
NULL concept (Is NULL a value or an attribute?), the 
results of MODIFY on derived data items not available to 
the Sub-schema, and the cascading effect of the ERASE 
ALL statement. Some of these problems may be eliminated 
by subsetting the database facility. 

CONCLUSION 

There are many obstacles in the path of the standardization 
of a database facility for the language COBOL. The Data¬ 
base Task Group X3J42 must develop a candidate Database 
module based on the CODASYL COBOL Journal of De¬ 
velopment 1978. The proposed module must then be ap¬ 
proved for publication for public comment as a draft pro¬ 
posed standard, either as a separate document or as a 
module of COBOL 80. The public comments are reviewed 


by X3J4, and changes made to the draft Database module 
based upon the public comments. The final Database module 
specifications must then be approved by letter ballot by 
X3J4 and their parent committee X3. After approval by X3, 
the Database Module is forwarded to the ANSI Board of 
Standards Review for approval of the publication of the 
Database module as an American National Standard. 

Even if a Database module is approved for inclusion in 
COBOL 80, much work is still required before there will be 
a database standard. The inclusion of a database facility in 
COBOL 80 would only standardize the COBOL Sub-schema 
and COBOL Data Manipulation Language statements. 
There would then exist the anomaly of a standard for the 
COBOL database facility without a standard for the Data 
Definition Language. Standardization of a COBOL database 
facility is meaningless without a Data Definition Language 
standard. 
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American national standards 
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INTRODUCTION 

The standards which interface with COBOL can effect the 
degree of machine independence a COBOL program has. 
COBOL 74 1 provides the capability to process the standard 
coded character set defined by the American National 
Standard Code for Information Interchange (ASCII). 2 In that 
regard, COBOL 74 relies on the ASCII Standard for defining 
the character codes and the collating sequence that are to 
be reflected by the program when processing that particular 
character code. Additionally, COBOL can specify that 
ASCII be recorded on external media such as punch cards 3 
or magnetic tape 4 with magnetic tape labels. 5 If that data is 
to be used for information interchange, then the recording 
of the data should be in accordance with the respective 
media standards. The American National Standards which 
interface with COBOL must be properly defined and imple¬ 
mented on a computer system for a COBOL program to be 
independent of the computer system. In the process of val¬ 
idating COBOL compilers, we use these other American 
National Standards related to COBOL and have found that 
data cannot be easily transported between computer sys¬ 
tems. 


ANSC X3 ROLE IN COBOL AND RELATED 

STANDARDS 

American National Standard Committee X3 through its 
three standing committees is responsible for the standard¬ 
ization of COBOL and other subjects related to systems, 
computer equipments, devices and media for information 
processing. 6 Three committees assist X3 in discharging its 
responsibilities concerning the administration, evaluation, 
allocation and scheduling of standards projects. The Stand¬ 
ards Planning and Requirements Committee (SPARC) and 
the International Advisory Committee (IAC) of X3 have 
staff responsibilities while the Standards Steering Commit¬ 
tee (SSC) has line responsibilities. The IAC is responsible 
for coordinating the work of ANSC X3 with respect to 
international organizations. The purpose of SPARC is to 
review the need for standards, make recommendations to 
X3 on the desirability of standards and determine that work 


of the technical committees is responsive to the original 
objective of the standardization project. The administrative 
arm of X3 is the Standards Steering Committee which mon¬ 
itors and coordinates the standards projects that are being 
developed by the project groups. Individual projects are 
classified as being in one of three project groups, hardware, 
software or systems. The technical committees (established 
for each individual project) designated for the development 
and maintenance of American National Standards for 
COBOL, character codes, magnetic tape labels, and data 
representation are part of the software group. Technical 
committees for the standardization of physical media asso¬ 
ciated with magnetic tape and punched cards are included 
in the hardware group. 

The basic objectives of the American National Standards 
Institute (ANSI) as well as its subordinate committees are 
the (1) promulgating of standards, (2) coordination of stand¬ 
ards development, (3) establishment of standards, and (4) 
exchange of information. The underlying purpose of the 
COBOL standard is to “promote a high degree of machine 
independence in such programs in order to permit their use 
on a variety of automatic data processing systems.”* Fur¬ 
ther, one of the criteria used by ANSC X3J4 (technical 
committee for COBOL standardization) for considering a 
COBOL element as a candidate in the 1974 COBOL stand¬ 
ard was its compatibility with other standards. A philosophy 
similar to COBOL is reflected in the standards associated 
with magnetic tape, punched cards, magnetic tapes labels 
and standard coded character set in that these standards are 
concerned with a common format for information inter¬ 
change between computer systems. Based on experiences 
noted later in this paper, there is a question as to whether 
COBOL and related standards, as yet are fully effective in 
providing information interchange. 


STANDARDS INTERFACING WITH COBOL 

There are five different American National Standards 
which may either directly or indirectly interface with Amer¬ 
ican National Standard Programming Language COBOL, 


* See Paragraph 1.1, page 1-1 in Reference 1. 
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X3.23-1974. The COBOL 74 Standard conveniently includes 
the language syntax which make these interfaces possible 
and thereby provides a common definition for the purpose 
of data interchange. American National Standard Code for 
Information Interchange (ASCII) is the common character 
code used for the interchange of data. Ideally, a standard 
data interchange on punched cards or magnetic tape is pos¬ 
sible between computer systems when ASCII together with 
the appropriate American National Standards for punched 
cards or for recorded magnetic tape and magnetic tape labels 
are used. For data interchange as well as program inter¬ 
change to be possible, a fundamental assumption is that 
each of the computer systems between which the data in¬ 
terchange is to occur must implement and support the ap¬ 
plicable ADP standards. 

Until COBOL 74, there was no definition within the 
COBOL language which allowed for the designation of a 
program collating sequence or how data would be repre¬ 
sented on external media. Without such a definition each 
implementor was permitted to use his own techniques re¬ 
garding the character collating sequence used, recording of 
signed data on external media, and the character code used 
for external media. The facilities in COBOL 74 for defining 
the collating sequence, the character code and representa¬ 
tion of signed data are the COBOL features associated with 
the PROGRAM COLLATING SEQUENCE clause, CODE¬ 
SET clause, alphabet-name clause and SIGN SEPARATE 
clause. For the first time, it is possible for COBOL programs 
to read, process and produce data that is interchangeable in 
accordance with related ADP standards. 


COBOL 74 


The COBOL language feature by which ASCII is written 
to or read from an external media is the CODE-SET clause 
plus an associated alphabet-name clause. The CODE-SET 
clause in an optional clause of the File Description Entry in 
a COBOL program. When it is present, this clause specifies 
the character code convention used on the external media. 
Any conversion that may be required of the code, as rep¬ 
resented internally in the computer system, and the code as 
represented on the external media occurs during execution 
of either an input or output operation (a READ or WRITE 
statement). The construct of the CODE-SET clause is 
“CODE-SET IS alphabet-name.” Alphabet-name is defined 
in the SPECIAL-NAMES paragraph and has the form: 


r STANDARD-1 ] 

NATIVE | 


alphabet-name IS 


implementor-name 

literal 


When the STANDARD-1 phrase is specified, the character 
code or collating sequence identified is that defined by the 
American National Standard Code for Information Inter¬ 
change, X3.4-1976. Each character of the standard character 
set is associated with a corresponding character of the native 


character code. Note the COBOL Standard does not re¬ 
quire that the computer system’s architecture use ASCII, 
only that the computer system and associated COBOL com¬ 
piler process the character code as if it were ASCII. 

The ASCII character code standard defines the bit con¬ 
figuration for representing characters in the standard data 
format. The value of numeric fields may be represented in 
either binary or decimal form depending on the equipment. 
If more than one type of numeric representation is available 
for a computer system, this selection may be specified 
through COBOL by use of the USAGE clause. The two 
options of the USAGE clause are COMPUTATIONAL (bi¬ 
nary) and DISPLAY (decimal). Therefore, in order to pro¬ 
duce ASCII on an external media via COBOL, data descrip¬ 
tions for the file must be explicitly or implicitly defined as 
DISPLAY. 

Another factor which effects the ability to interchange 
data between systems is the representation of signed data. 
The positive or negative characteristic of a numeric value 
may have, as many possible representations. Signs may be 
represented as part of the bit configuration associated with 
the high order or low order position of the numeric value. 
Signs also may be represented as a separate character which 
may be either leading or trailing. Since the ASCII character 
code does not provide for signed data values, the sign char¬ 
acteristic of a numeric value must be carried as a separate 
character. This is defined through COBOL using the SIGN 
SEPARATE clause. 

Data is only interchangeable if the sequence of data is 
characteristic of the character code selected and the COBOL 
program processes it accordingly. This sequence is con¬ 
trolled within the COBOL program by use of the PRO¬ 
GRAM COLLATING SEQUENCE IS alphabet-name 
clause located in the OBJECT-COMPUTER paragraph. The 
PROGRAM COLLATING SEQUENCE clause is optional, 
but if specified, the program uses the collating sequence 
defined by a corresponding alphabet-name clause. If the 
alphabet-name clause specifies the ASCII character code 
(alphabet-name option STANDARD-1) then all nonnumeric 
comparisons for any relation conditions during program ex¬ 
ecution should reflect the ASCII code collating sequence. 


AMERICAN NATIONAL STANDARD CODE FOR 

INFORMATION INTERCHANGE (ASCII) 

ASCII is the standard coded character set to be used for 
information interchange among information processing sys¬ 
tems. The ASCII Standard describes the code insofar as the 
bit arrangement or configuration relates to an associated 
graphic character. A minimum of 7 bits is used to represent 
an ASCII code character. The Standard does not define 
the means by which the code set is to be recorded in any 
physical medium but there is a bit relationship with the 
ASCII code to other standards such as the American Na¬ 
tional Standard Recorded Magnetic Tape for Information 
Interchange, X3.39-1973 (1600 CPI, PE) and also X3.22-1973 
(800 CPI, NRZI). For COBOL to produce or recognize 
ASCII data, the language feature CODE-SET in combina- 
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tion with the STANDARD-1 option of the alphabet-name 
clause is required. 

The binary bit pattern of the character code gives a col¬ 
lating sequence position for each character in the code. For 
an implementation to support the ASCII standard, the com¬ 
puter system must be capable of not only accepting and 
producing ASCII, but also must be able to process ASCII 
coded data in the proper collating sequence. 


AMERICAN NATIONAL STANDARD HOLLERITH 

PUNCHED CARD CODE X3.26-1970 

The Hollerith Punched Card Code Standard specified 256 
Hole-pattern in twelve-row punched cards. These Hole-pat¬ 
terns are assigned to the 128 characters of 7-bit ASCII and 
to 128 additional characters for use in 8-bit coded systems. 
This standard relates these punched Hole-patterns to a par¬ 
ticular bit pattern in an 8-bit data system. An example of 
such a system noted in this standard is the American Na¬ 
tional Standard Recorded Magnetic Tape for Information 
Interchange (800 CPI, NRZI), X3.22-1973. In addition to 
relating Hole-patterns to a particular bit construct, the code 
table presented in this standard also associates a graphic 
character to that particular position within the table. 
Columns 0-7 of the table correspond to the bit pattern and 
associated graphic character of that defined in the ASCII 
standard. The standard does not specify a card sorting se¬ 
quence; however, when we consider the bit patterns of any 
two ASCII characters there is a well-defined sequence re¬ 
lationship. 

Since the CODE-SET clause gives COBOL the capability 
to specify the character code convention used to represent 
data on the external media (such as punched cards) a rela¬ 
tionship does exist through the ASCII character code stand¬ 
ard for a COBOL program to accept, process, and create 
data on punched cards in accordance with X3.26-1970. 


AMERICAN NATIONAL STANDARD MAGNETIC 
TAPE LABELS FOR INFORMATION 
INTERCHANGE, X3.27-1976 

The magnetic tape label standard gives specifications for 
the recording of magnetically recorded labels on magnetic 
tapes used for information interchange. Rather than specify 
how the labels of the tape are to be processed or the data 
bit structure on the tape, this Standard identifies the infor¬ 
mation and the structure of tape labels that are recorded on 
tape. As such, the tape label standard is applicable to either 
7 track or 9 track magnetic tape. 

The standard makes note of the fact that the physical 
recording size of labels may be larger than the required 80 
characters subject to hardware constraints. These 80-char¬ 
acter records are grouped together based on their respective 
locations on tape related to beginning of tape volume, be¬ 
ginning of file, end of the file and end of tape volume. A 


tape mark, both before and after the file data, separates a 
label record group from the file data. Each label record is 
uniquely recognized by the first four characters in that rec¬ 
ord. A record group may have one or more records depend¬ 
ing on the implementation and an agreement between the 
data interchange parties. For a single volume tape all label 
records are optional except for label records identified as 
VOL1, HDR1 and EOF1 which respectively define the be¬ 
ginning of the volume or tape, header label for beginning of 
each data file, and label record immediately following the 
data file. An illustration follows: 

VOL1—HDR1—*File Data* + 

EOF1—** 

The tape label standard identifies the information and the 
data category (numeric or alphanumeric) that is to be stored 
in each field of the label record. ASCII is mentioned by the 
standard as a character code which can be used; however, 
the standard is applicable to any code structure. Certain 
fields of the label records are designated as optional by the 
standard. These optional fields may be recognized and proc¬ 
essed or may be ignored at the option of the implementor. 

COBOL includes provisions for specifying labels for data 
files through the LABEL RECORD STANDARD clause of 
the File Description Entry. However, COBOL 74 does not 
describe the format, structure, or content of file labels. It 
states only that file labels will be present when the clause 
is specified. If an implementation claims support of the tape 
label standard for use with COBOL, then the implementor 
should be expected to accept and process ANS tape labels 
on input, process those tape labels and create ANS tape 
labels on output, both in accordance with the magnetic tape 
label standard, X3.27-1976. 

AMERICAN NATIONAL STANDARD RECORDED 
MAGNETIC TAPE FOR INFORMATION 
INTERCHANGE (800 CPI, NRZI), X3.22-1973 AND 
(1600 CPI, PE), X3.39-1973 

The standards X3.22-1973 and X3.39-1973 for recorded 
magnetic tape provide specifications for the format and re¬ 
cording of half inch, 9-track magnetic tape to be used for 
information interchange with systems and associated equip¬ 
ment utilizing the ASCII code, X3.4-1976. These standards 
define the physical requirements for magnetic tape and the 
bit pattern for the recording of information on 9-track tape. 
Seven of the nine tracks are used for data; one track is used 
for character parity; and one track is treated as a bit of 
higher order than ASCII bits (shall be a zero bit). Each of 
the tracks used for the recording of data are assigned a 
particular bit identification which corresponds to the bit 
assignment used by the ASCII character code standard. 

COBOL 74 does not reference the recorded magnetic tape 
standards; however, it includes COBOL syntax and general 
rules governing the semantics for the COBOL statements 
which apply specifically to the treatment of files on magnetic 


f Each “*” represents a tape mark. 
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tape. Within COBOL 74, Sequential 1-0 module statements 
such as the CLOSE REEL, CLOSE WITH NO REWIND, 
OPEN WITH NO REWIND and OPEN REVERSED are 
used to specify the actions for files stored on tape or anal¬ 
ogous sequential media. Also the general rules associated 
with the READ and WRITE verbs define the actions to be 
taken when an end of reel is recognized during a read or 
write. 


USING COBOL WITH RELATED STANDARDS 

Programs written in compliance with the 1974 COBOL 
Standard language specification do not necessarily accept or 
produce data files that comply with the other data processing 
standards on many systems. Part of our work, here at the 
Federal COBOL Compiler Testing Service, involves the val¬ 
idation of COBOL compilers with respect to Federal Stand¬ 
ard COBOL 8 (ANS, X3.23-1974 COBOL). Because we are 
continuously implementing the COBOL Compiler Validation 
System (CCVS) 9 on different computer systems, the under¬ 
lying purposes for which standards are developed are of 
vital importance to our operations. More importantly, we 
test COBOL compilers and as part of those tests we validate 
the compiler’s capability to read, process and write ASCII 
in accordance with the 1974 COBOL Standard. 

Briefly stated, the 1974 CCVS consists of some 250 
COBOL programs which collectively test a COBOL com¬ 
piler’s conformance to COBOL 74 language syntax and se¬ 
mantics. That part of the CCVS used to validate the com¬ 
piler’s ASCII capability consist of three COBOL programs, 
an ASCII-encoded punched card deck, and an ASCII-en¬ 
coded magnetic tape. The ASCII validation requires three 
steps. In the first step, the first of the three-program set 
produces an ASCII-encoded magnetic tape and punched 
card file. The two remaining COBOL programs read and 
check the files produced by the first COBOL program. For 
the second step, the two programs which read and check 
the ASCII-encoded files are again executed. However, for 
this run they use as inputs an ASCII-encoded tape file and 
punched card file produced on another computer system. 
The purpose of this run is to validate that the compiler/ 
computer system being tested is capable of processing 
ASCII (represented both on magnetic tape and punched 
cards) that was produced by another computer system in 
accordance with the appropriate American National Stand¬ 
ards. The final step is to take the magnetic tape and punched 
card file produced during the validation to a different com¬ 
puter system for further checking. The purpose of this latter 
step is to determine whether the compiler/computer system 
being tested can also produce ASCII (represented both on 
magnetic tape and punched cards) which is capable of being 
processed on another computer system. 

Since COBOL 74 is not definitive regarding labels that are 
used for magnetic tape, the labels used for the ASCII portion 
of the compiler validation are dependent on the system 
under test. If the system under test supports the ANSI tape 


label standard, then tape labels conforming to ANS X3.27- 
1976 are placed on the magnetic tapes. If the system under 
test does not support ANSI tape labels then the magnetic 
tapes for the validation were supplied without tape labels. 

To recognize a system under test as being in full support 
of COBOL and associated data related standards, the system 
is expected to do the following: 

(1) Compile and execute the audit routines (CCVS pro¬ 
grams) that conform to the 1974 COBOL Standard 
(ANS X3.23-1974) without requiring any modifications 
to the source code, 

(2) Pass all the semantic tests made by the audit routines 
as reflected in the execution report of each audit rou¬ 
tine, 

(3) Accept ASCII-encoded (ANS X3.4-1976) data files 
from tape and punched cards that conform to the Mag¬ 
netic Tape Standard (ANS X3.22-1973 and ANSI 
X3.39-1973) and Hollerith Punched Card Code Stand¬ 
ard (ANS X3.26-1970) respectively, 

(4) Produce ASCII-encoded data files on tape and 
punched cards that conform to the same standards 
noted in 3 above, and 

(5) Accept and produce magnetic tapes containing mag¬ 
netic tape labels that conform to Magnetic Tape Label 
Standard (ANS X3.27-1976). 

Very few of the compiler/computer systems validated to 
date were successful in meeting all of the above criteria. 
The type of problems that were found follow: 

(1) Date recording formats unacceptable. One system 
tested resulted in an end of file condition occurring 
after processing the first record of a 50 record tape 
file. Another system could not read either a labeled or 
an unlabeled magnetic tape. However, this same sys¬ 
tem, for some unexplained reason, was able to create 
a tape that, when read on another computer system, 
proved to be correct according to applicable stand¬ 
ards. 

(2) Data code not read or recorded correctly. One system 
tested produced records on tape which included an 
additional character located in the first character po¬ 
sition of each record thus causing the contents of the 
record to be offset by one character. Another system 
did not provide proper character translation of all 
ASCII characters when reading punched cards. 

(3) Incompatible Tape labels. Some systems tested sim¬ 
ply did not support ANSI tape labels while others 
required the presence of optional labels HDR2 and 
EOF2. 

(4) Non-standard COBOL syntax. One system tested re¬ 
quired that the nonstandard clause “RECORDING 
MODE IS STANDARD-ASCII” be added to the SE¬ 
LECT Entry for all tape files. (The CODE-SET clause 
in the File Description Entry should have accom¬ 
plished this function). One system, oddly enough, 
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could properly read and write ASCII on punched cards 
only when the CODE-SET clause was removed from 
the program. 

A number of special considerations were necessary in 
order to process ASCII. Some implementations required 
that specific hardware models of magnetic tape and card 
readers/punches be used. Others required that specific op¬ 
tions be invoked through the job control language, and still 
others required particular system generation parameters be 
used when the operating system is generated. 


IMPACT OF COBOL 74 REVISION WITH REGARD TO 

OTHER STANDARDS 

In looking to the future to see how the revision to COBOL 
74 might affect the interfaces of COBOL with related stand¬ 
ards, it appears that there will be little or no impact. In a 
sense this is unfortunate because the COBOL standard could 
give more guidance as to the type of file labels that are to 
be used. The only feature for defining file labels within 
COBOL is via the LABEL RECORD STANDARD clause, 
which in COBOL 74 merely requires that labels will be 
present and conform to the implementor’s label specifica¬ 
tions. 

The difficulty most often found in transporting magnetic 
tape files between computer systems, next to improper im¬ 
plementation of standards related to COBOL, was the tape 
label. Many COBOL compiler/computer systems tested did 
not support ANSI magnetic tape labels. If the COBOL in¬ 
terface mechanisms for ANSI magnetic tape labels were 
similar in detail to the interfaces now present for ASCII, 
COBOL would have a direct link to another American Na¬ 
tional Standard. 


CONCLUSION 

One thing became quite apparent while testing the interfaces 
to ASCII and other standards related to COBOL. Even 
when strict adherence to American National Standards is 
maintained, COBOL programs and associated data files are 
not easily interchangeable among computer systems. This, 
in part, is due to improper implementation of ADP Standards 
by the parties involved in the interchange. Another factor 
is that the goals and objectives of X3 do not emphasize the 
compatibility of related standards or encourage interaction 
of related standards. 
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INTRODUCTION 

One property of the Data Encryption Standard (DES) is that 
each bit of ciphertext is a complicated function of all plain¬ 
text bits and all key bits. A method is developed which 
evaluates how fast this dependence (defined as intersymbol 
dependence) builds up as a function of repeated mathemat¬ 
ical operations called “rounds.” It is shown that the mini¬ 
mum number of rounds to achieve intersymbol dependence 
for plaintext as well as key is five. 

Cryptography deals with the methods involved in prepar¬ 
ing cryptograms—messages or writings intended to be in¬ 
comprehensible to all except those who legitimately possess 
the means to recover the original information. The designer 
of a cryptographic system, or cryptosystem, is a cryptog¬ 
rapher. 

The opponent or antagonist of a cryptosystem is a crypt¬ 
analyst. Cryptanalysis is concerned with techniques used 
to penetrate communications and recover the original infor¬ 
mation by means other than those available to the legitimate 
recipient. 

Cryptology is the science of disguised or secret commu¬ 
nications. It embraces both cryptography and cryptanalysis. 

The basic challenge in cryptography is to devise a method 
that transforms messages (known as plaintext) into crypto¬ 
grams (known as ciphertext) in a cryptographically secure 
way—that is, the method must withstand intense efforts of 
cryptanalysis. Plaintext can be protected by either of two 
techniques: it can be encoded using a code system, or it can 
be enciphered (encrypted) using a cipher System. 

Code systems require a code book or dictionary that re¬ 
lates the words, phrases, and sentences of the vocabulary 
(the plaintext) to its equivalent code group (the ciphertext), 
and vice versa. The number of plaintext messages that can 
be encoded depends on the number of combinations of 
phrases that can be obtained from the code book. Although 
that number may be large, not every combination or pattern 
of bits can be encoded. Hence the versatility and usefulness 
of code systems is limited, especially in computer applica¬ 
tions, which would have to be programmed to handle all such 
exceptions. 


Cipher systems require two basic elements: a set of rules 
or steps—that is, an algorithm—constituting the basic cryp¬ 
tographic procedure, which is constant in character and 
agreed upon in advance, and a specific cryptographic key, 
selected from a large set of possible keys, which is known 
only by the communicators. 

A cryptographic algorithm can be represented as an ex¬ 
tremely large number of possible mathematical procedures 
called transformations. Each transformation defines how se¬ 
quences of intelligible data (representing a message) are 
changed into sequences of apparently random noise (gib¬ 
berish) that are unintelligible to humans or machines. The 
cryptographic key is a secretly held sequence of numbers of 
characters, usually very short, which identifies the transfor¬ 
mation to be used. 

To be useful, the algorithm should have, for each of its 
transformations, a unique inverse operation that changes the 
gibberish back into intelligible data. (This process is called 
decipherment or decryptment.) However, this can be done 
only if one has exact knowledge of the key that was used in 
the initial encipherment (transformation). A cryptographic 
algorithm will, therefore, provide data security between two 
nodes of a data processing system if those two nodes have 
the algorithm installed (in hardware or software), and if both 
nodes have exact knowledge of the key. 

It is important to note that for good data security, only the 
key need be kept secret. The details of the algorithm are 
assumed to be known to everyone. The cryptographic key, 
therefore, is often considered analogous to the secretly held 
combination number of a safe. 

It is always possible to construct an unbreakable system 
if the number of characters in the key is equal to or greater 
than the number of characters of plaintext to be enciphered. 1 
It is required, however, that the key be randomly selected 
and used only once. This approach is impractical in data 
processing systems because of the large amount of message 
traffic. In a practical system, the key must be of fixed length, 
relatively short, and capable of repeated use. 

Basically there are three general classes of ciphers: trans¬ 
position ciphers, substitution ciphers, and combinations of 
these called product ciphers. A transposition cipher involves 
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the rearrangement or permutation of the plaintext symbols 
without change in their identity. A substitution cipher in¬ 
volves the replacement of plaintext symbols by one or more 
symbols without changing their sequence. Cryptographic 
research conducted during World War II showed that strong 
encryption systems could be obtained using alternate steps 
of substitution and transposition, resulting in a product 
cipher. 1 

Further research into the development of strong product 
ciphers was not undertaken in the private sector until the late 
1960’s. During the period from 1968 to 1975, a cryptographic 
procedure consisting of 16 alternate steps (or rounds) of key- 
controlled substitution and fixed permutation, based on work 
done by Horst Feistel, was developed at IBM. 2 This algo¬ 
rithm was accepted as a standard by the National Bureau of 
Standards on July 15, 1977, and is known as the Data En¬ 
cryption Standard (DES). 3 

Next in importance to having a strong algorithm is the 
incorporation of the cryptographic device into the system. 
In a weak implementation, by manipulating certain crypto¬ 
graphic functions, it may be possible for an opponent to obtain 
the desired information. For an example of a strong system 
implementation of the DES, see Reference 4 from which this 
introductory material was taken. 

Furthermore, if the key required for decrypting data is lost 
or unknown, the data cannot be recovered from the ci¬ 
phertext. It is just as difficult for the authorized user to 
decrypt these data when the key is unknown as it is for the 
opponent. 

The Data Encryption Standard’s principle of operation 
(without initial and final permutation on input and output, 
respectively) is shown in Figure 1. For strength, the DES 
relies on (1) the complexity of the function g, which incor¬ 
porates substitution as well as transposition 2 (or permuta¬ 
tion), and (2) exercising the function g a sufficient number 
of times (defined as rounds). 

After 16 rounds are employed by the DES, the 64 bits of 
plaintext are transformed into 64 bits of ciphertext under the 
control of a 56 bit key. During each round a subset of the 
key [defined as K(i) for round i; 1=1,2,. . . ,16] is used. 3 The 
input to round i, X(i- 1), is expressed as the concatenation of 
two 32 bit quantities: 

L(i— 1) represents the 32 input bits to the left half of the one 
round cipher operation (see Figure 1) and R(i— 1) the 32 input 
bits to the right half, i.e., 

L(/-l)= 4(i-l), 4(/-l), . . . , / 32 (i-l) 

R(i- l)=r t (/— 1), r 2 (i- 1), . . . , r 32 (?-l) 


Input = L(0), R(0) 



Figure 1—Basic block cipher design used by the data encryption standard 


bit block of output data (ciphertext), and K a 64-bit cryp¬ 
tographic key (eight bits may be used for parity checking, 
hence are not used by the algorithm itself). The notation 
f K (X)=Y means that the encipherment (/) of X under key 
K is equal to Y, and the notation f~ 1 K (Y)=X means that the 
decipherment (/ _1 ) of Y under key K is equal to X. 

Enciphering the plaintext 


X= 1000000000000001 

with a key 

A=3001010101010101 


results in 


Y= f K (A)=958E6E627A05557B 

where hexadecimal notation is used. 

If the same key is used to decipher this quantity, X is 
recovered. 

However, by letting 

Y* = 85 SE6E627A05557B 


instead of Y be deciphered with K, where Y* differs only in 
1 bit from Y, the result is 

f-\{Y*) =8D4893C2966CC21 

which shows the drastic effect of only a single bit ciphertext 
change on the decipherment process and, hence, indicates 
how strong a ciphertext bit depends on all plaintext bits. 

If, on the other hand, a key 


Hence the input to round (0 is defined as 


A* =1001010101010101 


X(i)=L(i), R(i). 

whereas the output from round (0 is defined as 
X(i-l)=L(i-l), R(i- 1), 

One property of the DES is that each bit of the produced 
ciphertext is a complicated function of all 64 plaintext bits 
and all 56 key bits. 

To give a representative demonstration of this property let 
X represent a 64-bit block of input data (plaintext), Y a 64- 


is used to decipher Y, where the 56 independent key bits of 
K* differ from K in only 1 bit, the result is 

f-\* ( y) =6D4£945376725395 

which shows the effect of only a one bit key change on the 
decipherment procedure and, in this case, indicates how 
strong a ciphertext bit depends on all key bits. 

The dependence of each ciphertext bit on all plaintext and 
key bits is defined here as “intersymbol dependence.’’ 
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There are many applications which can take advantage of 
this. 5 Two applications, which are demonstrated below, ef¬ 
fectively expand the block size of the DES by using chaining 
methods. When block encryption is used, each 64-bit block 
of data is enciphered separately. In chained block encryption 
on the other hand, the encipherment of each block is also 
made dependent on prior information (plaintext, ciphertext, 
or the like) that is available when the block is enciphered. 
Two important chained block encryption techniques are 
ciphertext feedback and plaintext-ciphertext feedback, as 
defined below. 

hetX 1 ,X 2 ,. . . , X n denote blocks of plaintext to be chained 
using key K, let Y 0 be a nonsecret quantity defined initial¬ 
izing vector, and Let Y x , Y 2 , , Y n denote the blocks of 

ciphertext produced. When ciphertext feedback is used, the 
following relationship holds: 

Y^fAX&Y^ ) for i>1 

where © represents module 2 addition. When plaintext-ci¬ 
phertext feedback is used, the following relationships hold: 

Y^MX&Yo) 

and 

Y^fAX&Yi- 10 ^-i) for /> 1. 

With blockchaining a ciphertext bit in block i therefore de¬ 
pends not only on all plaintext bits in block i but also on the 
plaintext bits of block 1 through (i—1). Thus chained block 
encryption (or block chaining) can be used to extend the 
intersymbol dependence between ciphertext and plaintext. 

Plaintext-ciphertext feedback has the additional property 
of error propagation. Corruption of a single bit of ciphertext 
will cause each subsequent bit of recovered plaintext to be 
in error with probability 0.5. By appending a known pattern 
of bits to the end of the plaintext prior to encryption, and 
comparing that value to the value recovered, the error prop¬ 
agation feature can be used for checking the true context of 
a message. Chaining techniques are useful for encrypting 
data, and the block cipher (with no chaining) is useful for 
key transformation operations. 5 

In the analysis to follow, a method is developed which 
shows how fast the intersymbol dependence builds up as a 
function of the number of rounds. The approach evaluates 
whether or not an output bit depends upon an input bit. 
Although the degree of complexity is not measured, a dis¬ 
tinction is made between three kinds of functional relation¬ 
ships. To discuss these relations, details of g (the kernel of 
the DES cryptographic approach) are illustrated in Figure 
2 . 

The right half of the input to round (0 is expanded from 
32 bits to 48 bits and is denoted by E[R(i— 1], using the E 
Bit-Selection Table. 3 The result is: 

E[/?(i-l]=r 32 ,r 1 ,r 2 ,r 3 ,r 4 ,r 5 ,r 4 ,r 5 ,r 6 ,r 7 ,r 8 ,r 9 ,/- 8 ,r 9 ,r 10 ,r 11 , 

*12 5 *13 5 *12 5 *13 5 *14 5 *15 5 *16 > *17 5 *16 5 **17 5 *18 5 *19 5 ^ ) 

*20 5 *21 5 *20? *21 5 *22 5 **23 5 *24 5 **25 5 *24 5 *25 5 **26 5 **27 ’ 

**28 5 **29 5 **28 5 **29 > *30 5 *315 **32 5 *1 

where the parentheses have been dropped for convenience 
of representation. The purpose of using the E expansion is 


to achieve a dependence of each ciphertext bit on all plain¬ 
text bits in as few rounds as possible. 

Next £*[/?(/■-1)] is added modulo-2 to K(i), resulting in a 48 
element vector, (A). 

A =E[R(i -1 )]©«*(/) 

— **1 , ^2 5 • • • , a 48 ■ 

[Although (A) is different for each round and should there¬ 
fore be indexed by (/), to avoid unnecessary details, it is not 
done here.] 

In Figure 2 the elements of vector (A) are used as argu¬ 
ments in the substitution operations (5-boxes) S i through 
5 8 . 3 Each 5-box is described as a matrix of four rows (labeled 
00, 01, 10, 11) and 16 columns (labeled 0000, 0001, . . . , 
1111 ). 

Each 5-box can now be represented by four substitution 
functions, i.e., 5, 00 , 5 4 01 , 5, 10 , 5* 11 for 5-box 5, where the su¬ 
perscript identifies the row of the matrix, each mapping four 
input bits (which determine the column of the matrix) to 
four output bits given by the element of the matrix. 

The first and last input of the six entries to the DES 
substitution box 5* in Figure 2 determine which of the four 
substitution functions in 5* are selected. Because these bits 
are derived from input bits (i.e., message bits if the input is 
interpreted as a message) as well as key bits, the selection 
of substitution functions not only depends on the key but 
also on the input data. Thus the E expansion, in effect, 
introduces an autoclave or self keying feature. Since (0 
ranges from one through eight, there are 32 functions that 
are obtained with the eight 5-boxes. 

With a x through a 6 being the inputs to the first 5-box, 

5 1 ° 1 ’ a6 (a 2 , Og, a 4 , %) 

represent the first four bits of ( B ), i.e., bits b x through b A . 



32 bits 32 bits 


Figure 2—Details of ( g ) enciphering function 
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The last four bits, from S 6 , i.e., b 29 through b 32 are given by 
S s a 43 ,aiH {a i4 , a 45 , @ 4 $ , < 347 ). 

Knowledge of the 48 bits represented by vector (A) there¬ 
fore allows the 32 bits represented by vector ( B ) to be 
determined. By permuting the elements of B, P(B ) is obtained 
as follows. 3 

P(B)=b l6 , b 7 , b 20 , b 21 , b 29 , b l2 , b 28 , b 17 , b x , b 15 , b 23 , 

b 38 , b 5 , b 18 , b 31 , b 10 , b 2 , b s , b 24 , b 14 , b 32 , b 27 , (2) 

b 3 , b 9 , b 49 , b i3 , b 30 , b 8 , b 22 , b n , b 4 , b 25 

The output of round (/) thus becomes 

L(i), R(i)=R(i- 1), P{B)®L{i-\) 

Furthermore, 

L(i)=R(i-l) (3) 

and 

R(i)=L(i- l)©g(*(i), R(i- 1)) (4) 

where the 48 bits of K(J) are a subset of the inputted key as 
previously discussed. 

In the following two analyses, the dependencies of output 
X(i) on plaintext X(0) and on the key are treated separately. 


INTERDEPENDENCE BETWEEN CIPHERTEXT AND 
PLAINTEXT 


To investigate the functional relationship between the 
input to the (/+l)-th round (which is equal to the output of 
the (i)-th round, i= 1, 2, 3, . . . , 16), and the input to the first 
round, a matrix is defined. Consisting of 64 rows and 64 
columns and referred to as G Uj , its element a Um for row l 
and column m shows the type of relationship which exists 
between the l-th bit of X(i) and the m-th bit of X(j). In 
particular, a Um is “blank” if a dependence does not exist 
between x x (/) and x m (J). If there is a dependence via message 
bits only, a Um is set to “x”. If the dependence is via auto¬ 
clave, a Um is set to ”. If message bits as well as autoclave 
bits influence the output, a Lm is set to “*”. 

In a well-designed system, an output bit depends on more 
and more input bits as the number of rounds increases. 
These input bits come into play via message (x) dependence, 
autoclave (-) dependence, or both (*). The design goal is to 
have each output bit depend on each input bit after only a 
few rounds, with autoclave as well as message dependence 
being achieved. 

It is advantageous to partition matrix G u into four sub¬ 
matrices of 32 rows and 32 columns each, i.e.. 



Using the definition of G u , the submatrices’ elements ex¬ 
press the relationships as shown in Table I. 

Let the relationships between the output and the input of 
one round, i.e., G iA - 4 be evaluated first. The dependence of 


TABLE I.—Functional Relationships 


Submatrix 

Relationship expressed in submatrix 

GJ L - L > 

L(i) vs U j) 

G u aj,) 

L(i) vs R(j) 

G u iRM 

R(i) vs L(j) 

G iiJ iRJt> 

R(i) vs R(j) 


the output bits from the substitution operation (vector ( B ) 
in Figure 2) on the input bits R(i~l), explained below, is 
shown in Figure 3. [The numbers identifying the row and 
column of the matrix also refer to the index of the element 
of the appropriate vector, i.e., ( B ) vs R O'- 1 ), respectively, in 
Figure 3. Furthermore, note that ( B ) vs R(i— 1) is identical to 
R(i) vs R{i-l) if the permutation is not present]. 

The selection of the substitution function (5-function) in 
the first 5-box, S lt depends on the 32nd and fifth bit of 
R(i— 1). This follows from the fact that (1) the first and sixth 
input bits to 5j select the 5-function, and (2) the first and 
sixth bits of the expanded version of R(i— 1), which is deter¬ 
mined by E[R ( 1 -1)], is equal to r 32 (i -1 ) and r 5 (/-1 ), respec¬ 
tively, according to Eq. 1. Therefore, all output bits from 
5 X , i.e., b\ through b 4 , depend on r 32 (/-l) and r 5 (i-1) via 
autoclave. However, they depend on r 4 (/— 1 ) through r 4 ( i - 1) 
via message. R(i—l) is considered to be the input message 
in this case. Hence, the entries in rows 1 through 4, columns 
32 and 5 in Figure 3 are equal to The corresponding 

entries in columns 1 through 4 are equal to “x”. All other 
entries of rows 1 through 4 are blank since b 4 through b 4 
only depend on six input bits. Repeating this analysis for 5 2 
through 5 8 results in the matrix of Figure 3. 

Taking permutation into account, the matrix rows have to 
be rearranged according to the permutation schedule given 
by P(B) in Eq. 2. The result is shown in Figure 4 and rep- 


R(i-1) 



1 5 10 15 20 25 30 


Figure 3—Functional dependence of R(j) on R(i-l) without permutation 
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- X X X X 


- X X X X 


- X X X X 
- X XXX- 


- X X X X 
X X X X - 


x x x x - 

- X XXX- 


X X X X - 


R(i) 


R(i-1) 

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 

-j - x x x x - 

2 

3 

4 

5 

6 

7 

8 

9 x x x x - 
10 
11 
12 

13 - x x x x - 

14 

15 - 

16 - x x x x - 

17 x x x x - 

18 - x x x x - 

19 - x x x x - 

20 - x x x x - 

21 - - x x x x 

22 - x x x x - 

23 x x * x - 

24 - x x x x - 

25 - x x x x - 

26 - x x x x - 

27 - - x x x x 

28 - x x x x - 

29 - x x x x - 

30 - x x x x - 

31 x x x x - 

32 - x x x x - 

Figure 4 —Functional dependence of R(i) on 7?(/-l), matrix G w _ t aiJl> 


- X X X X 


resents the type of relationship that exists between R(i) and 
R(i- 1); thus, G u _ 1 (RJl) is obtained (see Table I). 

From L(i)=R(i-l), see Eq. 3, L(i ) does not depend on 
L(i- 1) but has a linear dependence on R(i- 1). Hence the 
elements of G isi _ 1 a>L) are blank since they express the rela¬ 
tionship between L(i) and 

The relationship between L(i) and R(i- 1) is expressed by 
the elements of G iA ^’ R) - Due to the linear relationship be¬ 
tween L(i) and R(i-l), as discussed before, the elements 
located on the diagonal are set to “jc” as shown in Figure 
5. Eq. 4 shows the relationship of R(i), L(i— 1), K(i), and 
R(i— 1). Since the dependence on K(i) is not of interest here, 
the expression 

R(i)=L(i— l)+/i[i?(i— 1)] (5) 


shall be used hereafter. The relationship, therefore, between 
R(i) and L(i— 1) is also linear and, hence, the diagonal ele¬ 
ments of G iA _ l (R,L) ‘ which relate R(i) and L(i— 1), are set to 
“jc”, whereas the remaining elements are blank, as shown 
in Figure 5. 

Having evaluated G u _ 1 a,I ' ) through G u _ 1 (Rjft) , the matrix 
G iA ^ is completely defined and is shown in Figure 5. 

Let be expressed next in terms of From Eq. 

3 the relation L(i+l)=R(i) can be established. Thus the 
functional dependence of L(i+ 1) on X(i- 1) is identical to the 
functional dependence of R(i) on X(i- 1). Since the depend¬ 
ence of L(i+ 1) on X(i-l)=[L(i-l), i?(i-l)] is given by 
G i+W _ 1 a ’ i) , G i+ui ^ LJt) , and the dependence of R ( i ) on X(i- 1) 
is given by G i)i _ 1 (ft * i) , G iA ^ RyR \ according to Table I, it fol¬ 
lows that 


and 


r: (l,l)— fi (r,l> 

KJ i+l,i-1 


( 6 ) 


a 


i+l.i—1 


(.L,R) = 


n own 


(7) 


Before showing how G i+W _ 1 ( *’ i) and G i+lA _ 1 (RJi) may be 
obtained, a more general case is considered, i.e., the eval¬ 
uation of G j+1>i _ 1 <ft>i) and G J + u _ 1 (flJl> ( j^i) from (Note 
that for j—i the above stated special case is obtained). 

The elements of row s of G j+lA ~ 1 <R,L) give the relationship 
between bit 5 of R(j+ 1), i.e., r s (y+l)and L(/-l), whereas 
the elements of row s of G j+1>i _ 1 (ft * fi) give the relationship 
between bit s of r(j+ 1) and R(i- 1). To evaluate these ele¬ 
ments the relationship between r s (j+ 1) and X(j)=[L(j), 
R(j )] is obtained first by using Figures 4 and 5. Bit s of 
R( j+ 1) linearly depends on L( j) as shown in Figure 5, i.e., 
on bit s of L( j ) via message dependence (x). From Figure 4 
it follows that bit s of R( j+1) depends on R(j) via -xxxx-, 
i.e., bit s depends on two bits of R( j ) via autoclave and on 
four bits of R(j) via a message relationship. Let the column 
locations of these entries in G iA ^ R,R) (equal to G j+1A R ’ R) if 
i= j+ 1) be designated as m 1 (s) through For exam¬ 

ple, using Figure 4, the following is obtained for 5=4: 
m 1 (5) =20, m 2 (s)= 21, n%{s)=22, m 4 (s)= 23, m 5 (s)=24, and 
m 6 (5)=25. Since the dependence of X( j) on X(i— 1) is given 
by G jA _ t and the dependence of bit s of R(j+l) on X(j) is 
known, the relationship between R(j+ 1) and X(i— 1), given 
by rows 32 to 64 of matrix G^^ can be determined by 
properly combining elements of row s, m^is) +32, . . . , 
m 6 (5)+32 of Gj A - 4 . This method is indicated graphically in 
Figure 6. 

The question still to be answered is, “What rule should 
be used to combine rows of matrix G iA so that the de¬ 
pendence of x s+32 ( j+1 ) on X(i- 1) can be evaluated?” Since 
the main objective of this analysis is to determine how fast 
the functional dependence between output and input builds 
up it would be sufficient to simply indicate if a functional 
relationship between an output bit and an input bit exists or 
not. However, to gain more insight on the influence of au- 


L(0 


R(i) 




< 

> 

< 




L(i-1) 

A 

R(i-1) 

_-__ 




Elements equal 

s 

\ 

s 

to “blank” 

\ 

\ 

G (L j L i 

\ 


1 . n s 


\ 

N 

\ 

\ 

\ 

N (R.L) 

S N G i,i-1 

^(R.R) 


N 

\ 

Elements shown 

\ 

in Fig. 4 

/ 

/ 

/ 

/ 

/ 

_ 


Elements of Gj. m and GV m located on the indicated diagonal 
are equal to “x” whereas the remaining elements are equal to blank. 


Figure 5—Functional relationship between X(i)=L(i), R(i) and 
AT(i—l)=L(i—1), matrix G Ui ^ 
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Note: X(j + 1) = |L(j + 1), R(j + 1))vs X(i-1) is obtained by 
combining the elements of row s, m, (s) through m 6 (s), 
where m, (s) through m6(s) are the columns in which the 
elements in row s of Gf R , R) occur. (1^s^32.) 

(R R) ’ 1 

G, M is defined in Figure 4 

Figure 6—Evaluation of functional dependence of X(j + 1) vs A'(i-l) from 
X(j) vs. *(/-!) 


toclave, a rule was devised which highlights the autoclave 
influence. Thus all elements of G jA _ t for rows (s )+32 and 
m e (s)+32 are set to indicating an autoclave relation¬ 

ship. Note that the elements of row 5+32 of G j+ui - t describe 
the dependence of x g+32 (./+l) on X(i-l). Since Xg +32 (j+l) 
depends on ^ l(g)+32 ( j) and j) via an autoclave re¬ 

lationship, the dependence of * mi ( g) + 32 (./)and ^ m6 ( S)+32 ( j) on 
X(i— 1) is changed from “jc” or to an autoclave 
dependence “ —By adopting this rule, previous history, 
as far as the autoclave entry to an 5-box is concerned; is 
eliminated, thereby increasing the emphasis on the last au¬ 
toclave dependence. 

In order to take into account previous history for the 
entries to an 5-box not associated with autoclave, the ele¬ 
ments of Gj.i-i are left unchanged for rows s, (s) +32, 
Ang(5)+32, m 4 (5)+32and m 5 (s)+32. 

In combining the proper rows of G 5A ^ one or more entries 
can occur in the same column. Only if entries are either all 
“x” or will the final dependence be correspondingly 
set. If mixed entries (“jc” and occur, the final de¬ 

pendence is set to “*”. If entry occurs at least once, 
then the final dependence is set to regardless of what 
the other entries are. 


Procedure summary 


To generate the elements of row (5+32) of matrix G j+lji _ 1} 
for 1^5<32 determine m 1 ( 5 ) through 7 ^( 5 ) from the given 
matrix G iA - 1 iRJt \ They are the columns in which the non¬ 
zero (i.e., non-blank) elements in row s of G iji _ 1 (B,B) are 


located (Figure 4 gives these entries). Take matrix G }A - t and 
select row 5, (5)+32 through m6(5)+32. The entries in row 

(5+32) of matrix G j+1A _ t are now obtained by combining the 
elements in the indicated rows using the established rules. 

As a corollary, the following can be stated: 


Elements in row 5 of G j+ljj _ 1 (B>i) are obtained by combining 
row 5 of G jjj _ 1 (Z " L) with rows m x (5) through (5) of G jA -^ R,L \ 
Elements in row 5 of G j+l i _ 1 (B * B) are obtained by combining 
row 5 of G jii „ 1 a>B) with rows m, (5) through mg (5) ofG jA _ 1 (RJt) - 


Let the special case j=i be considered now, i.e., let 
G i+U _! be evaluated from G u _ x . Using the stated rules, the 
submatrix G i+lji _ 1 (fi * B) can be derived. The result is shown in 
Figure 7. 

Furthermore, it can be shown that 


using an approach similar to the one applied in the proof by 
induction shown below and hence (from Eq. 7) 

(-1 (R,L)—r2 (L,R)—n (R,R) 

<jr i+l,i-l '- 7 t+l,i —1 \p) 


Using also Eq. 6 the evaluation of G i+lA ^ 
G w _j can be completed. 

in terms of 

The proof by induction is used next to show that the 

following general rule (for j^i) holds: 


a <«+> 

(9) 

ri (( RJt ) 

”‘+1,1-1 ”,1—1 

(10) 

(-1 (RJ.)-n (LJD—ri (RJ {) 

Vl+l.i—1 ”+1,1—1 ”,i— 1 

(ID 


The first stage of the proof was already completed by 
showing that these statements are correct for the smallest 
value of j,j=i. (See Eqs. 6, 7 and 8.) 


2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

R(i+1) 15 
16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 


RO-1) 

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 

* --*XX**XXX- — X X XX— -----*xx* 

- * - - - -XXXX- -x XX * - - — - * X X * * X X * 

*XX**XX*-- - * X X X - —XXXX— ----- 

¥ X X X - -XXX*--- — X X X * * x X X 

XXX**---*XX*- — X XX **XX X- — 

XXX*-*--- -XXX*---*X X * * X X X — 

-XXX* — — *X X** X X *— — — — — -XXXX 

*xx**xx* -— — * X XX— -XXXX- -- - - 

-XXXX* -X XX**X x**XX*------ 

- — — — * x X * * X X X— -XXXX- --- * x X * 

* X X X - -XXXX----- -xxx**xx* 

-X XX *---**X X**X X*--- -XXXX 

--- -XXXX* — X XX* — — — — *xx**x X* 

*XX**XX*-- * X X X - -XXXX- -- 

XXX*--* X X * - - *- — xxx**xx X — 

XXX*-- -XXX*--*XX**XXX — 

— XXXX- -X XX** xx**xx*--- --- 

-- -XXXX- -XXX*- --*x X * * x X * 

* X X X - -XXX*----*- — X X X * * X X * 

----* X X** XX X- -XXXX- — — — — — * X X* 

xxx*----*xx*-- -X XX**X X X- - 

-XXX*-— — * X X * * X X * — if. — — — -XXXX 

-XXXX- -X XX** XX** XX*—--- 

XXX*--- — X x x* — — — — *X X**XX X- - 

JfXX**X X*-— * X X X — -XXXX* ---- 

--*XX**XXX- -XXXX- --*-_*xx* 

X XX*— — — — *xx *- - -- — X XX **XX X — 

-- -XXXX- — X XX* — — — — *XX * * X X * 

* X X x - -XXX*----- — *X X **X X* 

XXX*---- -X XX * - - -- *xx* *xxx-x 

- -XXXX- -XXX**XX**XX*---- * - 

- -XXX*- -*xx**xx*- - -XXXX 


Figure 7—Functional dependence of /?(»+!) on /?(/-!), matrix 
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From the relationship (Eq. 3) 

L(j+l)=R(j), 


it can be determined that L(j+ 1) depends onl(i-l) in the 

same way R(j) depends on X(i— 1). Thus, 

r: (UD—rz ot,L) 

a (l,r) = cl out) 


which proves Eq. 9 and 10. 

To prove Eq. 11 the results stated in the corollary are 
used as follows: 


G j+ui _ l <R - L) is obtained by properly combining the elements 
of G Jji _ 1 a,i) (row s) and (rows m, ( 5 ) through n%(s)). 

But G w _ 1 a ’ i) =G j _ u _ 1 (fl ’ L) , G jti _ 1 (R,L) =G j _ 1>i _ 1 (RJl \ if the stated 
rule holds. 


Gj +1) i_ 1 a * iJ) =G j(i _ 1 (R,R) is obtained by properly combining the 
elements of G j _ lti _ 1 tt * 8) (row s ) and G J _ U _ 1 08 ’ B) (rows raj (5) 
through m^(s) ). But G j _ w _ 1 a " S) =G i _ 1>i _i ( ®’ L) if the stated rule 
holds. 


Hence, G j+1;i _ x (jBiL) is obtained in an identical way as G M _ 1 (fl ’ fl) 
and, likewise, G i+l i _ 1 (Z " fl) is obtained. Therefore, G J+u _ 1 (fl ’ i> 
=G j+ui _ l <L ’ R) which completes the proof (see Figure 8 for a 
graphical representation). 


Minimum required number of rounds to achieve ciphertext / 

plaintext intersymbol dependence 

For each output bit to depend on all plaintext bits after 
round (j), no element of G j>0 can be blank. When this con¬ 
dition is satisfied, it follows from the developed relationships 
between G j>0 and G,_ ls0 , Gj_ 2 , 0 that (1) no elements of 
Gj-if L ’ R) =G i - l ' 0 iR ’ L) can be blank and (2) no elements of 
Gj- 2 , 0 (R ‘ R) can be blank. 

By assuming that all row entries of the matrices are in¬ 
dependent, an approximate result for the minimum number 
of rounds can be obtained, as shown below. (In actuality the 
elements of eight sets of four rows each of G W _ 1 (R * S) are highly 
correlated. Therefore, the buildup of intersymbol depend¬ 
ence will be slower than predicted by the approximation.) 

Since G 1>0 (RJt) has six entries (Figure 4) in each row and 
G u f L,R) has one entry in each row (Figure 5), then G 2 f RJt) 



Figure 8—Graphical presentation of proof that G J+1>i _ 1 <R,L) =G J+lji _ 1 l 


can have at most 1+6-6 or 37 entries. (Note that G 2 f R,R) is 
obtained from G li0 (LJi) and G 1 f RJ{) .) There are, however, only 
32 columns in G 2 f R ’ R \ Therefore, it is possible that none of 
the elements of G 2 f R,R) are blank. Thus, it is possible to fill 
all 32 2 entries of G 2 ’ 0 (RJ{) , G 3 f LJt) =G 2 f R ^\ G 4>0 (fl * B) after rounds 
2, 3 and 4 respectively. Hence, all 64 2 entries of G 40 could 
be filled after four rounds. 

Thus the approximate analysis shows that four is the min¬ 
imum number of rounds to achieve intersymbol dependence 
between ciphertext and plaintext. The accurate analysis, 
performed below, shows that after four rounds complete 
interdependence has not been achieved and for the specific 
design of the DES a minimum number of five rounds is 
required. 

From the evaluation of G i+u _ 1 (fl - fi) (Figure 7) there are 28 
non-blank elements in each row, except for row 30 which 
contains 31 elements. It can also be shown that all entries 
of G i+2>i -f R,R) are equal to Thus, after Round 4 each 
output bit of L( 4), except bit 30, depends on 28 bits of L(0) 
and all 32 bits of i?(0). Bit 30 of L( 4) depends on 29 bits of 
L( 0) and all 32 bits of /?(0). Each bit of R(4), on the other 
hand, respectively depends on all 32 plaintext bits of L(0) 
and R(0). 

Let the degree of intersymbol dependence be measured 


TABLE II.—Ciphertext/Plaintext Intersymbol Dependence 


Round 



Output/Input Relation 



j 

L(j) vs L(0) 

L{j) vs R( 0) 

/?( j) vs L( 0) 

R(j) vs R{ 0) 

XU) vs *(0) 

1 


3.13 

3.13 

18.75 

6.26 

2 


18.75 

18.75 

88.18 

32.20 

3 

18.75 

88.18 

88.18 

100.00 

73.78 

4 

88.18 

100.00 

100.00 

100.00 

97.05 

5 

100.00 

100.00 

100.00 

100.00 

100.00 


NOTE: Table entries express the degree of intersymbol dependence, i.e., the percentage of non-blank 
elements in the appropriate relation. 
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by a factor, £, which is obtained by evaluating the percent¬ 
age of non-blank elements. Respectively, for G 1 , 0 u " i) through 
G 1>0 (RJi) (Figures 4 and 5), £ is 0; 100-32/32 2 = 3.125; 3.125; and 
100-32-6/32 2 = 18.75 percent. With the total G 10 , ij is equal to 
1/4 (0+3.125+3.125+18.75) or 6.25 percent. To obtain £for 
G 2 /™\ either Figure 7 or the previously quoted results can 
be used to arrive at 100-(32 2 (31-4- 3))/32 2 or 88.18 percent. 
Hence, using the rules obtaining G s o from G u0 , £ associated 
with G 2t0 (L>L) through G 2S) (R ’ R) is 3.13 percent, 18.75 percent, 
18.75 percent, 88.18 percent, respectively, whereas £ for G 
2,0 is equal to 1/4 (3.125+18.75+88.18) or 32.20 percent. 
Because none of the elements of G 3 , 0 iR,m are blank (they are 
actually all “*”) £, as associated with G 3 , 0 (R,R) , is equal to 
100 percent. The results are summarized in Table II. 


INTERDEPENDENCE BETWEEN CIPHERTEXT AND 

KEY 

The basic ideas developed in the previous analysis can be 
used to investigate the intersymbol dependence between the 
output of the O'-th) round and the 56 bit cryptographic key. 
However, since the keys used in the different rounds vary, 
the analysis is slightly more complicated. Since the condition 
L(j+l)=R(j) introduces a certain symmetry, it is advan¬ 
tageous to evaluate the dependence of L( j ) and R(j) on the 
key in addition to the dependence of X( j) on the key. The 
details of the analysis are shown in Reference 5 and the results 
are quoted in Table III. 

It is concluded that the minimum number of rounds is equal 
to five to achieve intersymbol dependence between cipher- 
text and key. 

SUMMARY AND CONCLUSIONS 

On July 15, 1977 an encryption algorithm, called the Data 
Encryption Standard (DES), became effective as a U.S. 
Federal Standard. The algorithm utilizes a secret cipher key 
(a sequence of bits) to transform 64 bits of plaintext into 64 
bits of ciphertext. 

One property of the DES is that each bit of ciphertext is 
a complicated function of all plaintext bits and all cipher key 
bits. This property, defined as “intersymbol dependence” 
was analyzed by evaluating how fast intersymbol depend¬ 
ence was achieved as a function of repeated mathematical 
operations which were defined as rounds. Each of these 


TABLE III.—Ciphertext/Key Intersymbol Dependence 


Round 

j 

Output/Input Relation 

L(j) vs Key 

R(j) vs Key 

X(j) vs Key 

1 

0.00 

10.71 

5.36 

2 

10.71 

79.02 

44.86 

3 

79.02 

96.43 

87.72 

4 

96.43 

100.00 

98.22 

5 

100.00 

100.00 

100.00 


NOTE: Table entries express the degree of intersymbol dependence, 
i.e., the percentage of non-blank elements in the appropriate relations. 


operations consists basically of substitution and transposi¬ 
tion. 

Three different forms of dependencies were considered. 
If an input bit affects the selection of a substitution function 
in one round, the corresponding output was said to have an 
autoclave dependence on the input. 

If an input bit affects the argument of a substitution func¬ 
tion in one round, the corresponding output was said to have 
a message dependence on the input. 

Since the basic operation involving substitution and trans¬ 
position is repeated several times, the functional relation 
between a ciphertext bit and plaintext bit can, in addition to 
the already defined relation, be a combination of both. 

It was shown that after five rounds each ciphertext bit 
depends on all plaintext bits via message as well as autoclave 
dependence. In addition, a similar analysis revealed that each 
ciphertext bit depends on all key bits after five rounds. 

The method applied to the DES is, in general, applicable 
to ciphers which (1) split up the input into two parts, (2) 
operate on one part first, and (3) combine the results of that 
operation with the other part using modulo 2 addition. 
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INTRODUCTION 

The concept of a data dependent key for use in conjunction 
with a selective encryption device is presented. The use of 
an encryption key extracted from the data or a function of 
extracted data eliminates the problems associated with a 
statistical assault on encrypted data which was all encrypted 
using the same key. If this key extraction is done in hard¬ 
ware then the security of the key is reduced to the physical 
security of the device which encrypts/deencrypts and not 
on the personnel doing the data entry. This work is premised 
on and builds from work done by Carson, Flury, Summers 
and Welsh, 1,2 establishing the concept of selective encryp¬ 
tion terminals as a feasible technology. 

With the passage of the Privacy Act of 1974, 3 a new area 
of concern has been presented to the data processing indus¬ 
try, i.e., the required protection of data on individuals. 

The recently published report on the Privacy Protection 
Study Commission 5 goes further. “. . . the ability to search 
through . . . records to identify individuals with particular 
characteristics of interest is at once the most important gain, 
and the most important source of potential harm, stemming 
from the automation of large scale personal-data-record- 
keeping systems, and its importance is likely to grow in 
importance as ‘textual search’ techniques are refined.” 6 

In light of this, various methods of protecting data have 
been suggested. Among these is the adoption by the National 
Bureau of Standards of an “Encryption Algorithm for Com¬ 
puter Data Protection.” The algorithm is designed to enci¬ 
pher and decipher blocks of data consisting of sixty-four bits 
under control of a sixty-four bit key. 4 This algorithm was 
implemented in an experiment performed by the METREK 
Division of the MITRE Corporation in 1975 and 1976. They 
developed the Selective Encryption Terminal which consists 
of a Tektronix 4023 terminal and an LSI-11 microcomputer 
in communication with a host computer where the database 
was stored. For this experiment the host was an IBM 370/ 
145 running the VM/CMS operating system. 1,2 


SELECTIVE ENCRYPTION 

The concept of selective encryption allows the terminal 
operator to encrypt only those portions of an individual’s 
record for which protection is required or to encrypt the 
identification portion of a record so that the individual is not 
associated with the data. The latter is useful in databases 
where statistical studies are made with respect to certain 
aspects of the individual’s files. Selective encryption also 
allows the terminal operator to change the key so that dif¬ 
ferent sections of an individual’s record can be encrypted 
with different keys. In the METREK experiment, a hospital 
database was used. Here, medical information was en¬ 
crypted under one key for use by physicians while account¬ 
ing information was encrypted under another key for use by 
the hospital’s administration. 

The most important feature of the Selective Encryption 
Terminal is best expressed in this excerpt from the 
METREK report. 

The protected material remains in encrypted form 
throughout its residence in the host system and is de¬ 
crypted only upon return to the terminal. In this manner, 
neither the protection (encryption/decryption) software, 
the protection key, nor the plaintext of the protected in¬ 
formation is ever present in the host computer . 1 

The plaintext referred to is the data before encryption. 


VULNERABILITY OF THE KEY 

It is important to note that the security of the METREK 
system and any other system utilizing the NBS algorithm is 
entirely dependent on the security of the key or keys. Given 
the information that the NBS algorithm was being used, an 
outside intruder could get access to the protected data by 
one of three ways. First, the key could be obtained by trial 
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and error: iterating through all possible values of the key 
until found. Second, the encrypted text could be analyzed 
statistically so that any patterns in the data might reveal the 
key. And finally and most simply, the key could be obtained 
from those personnel who are exposed to the key, such as 
terminal operators. 

What is to be considered then, are variations in the im¬ 
plementation of the algorithm which would combat these 
problems. One such variation is the concept of using data 
dependent keys in performing the encryption. Discovering 
the key by trial and error even when the data dependent 
keys are not used is a difficult task. With an eight byte key, 
each byte an ASCII character in which only seven of the 
eight bits vary, there are fifty-six bits which numerate all 
possible keys, and if it were possible to process each attempt 
in one microsecond, it would take on the average over one 
thousand years to find the key. 7 The problem occurs if the 
intruder trying to break the system gets lucky or utilizes 
some fast and advanced multiprocessing system. With the 
key found, the entire database is at his disposal. 

DATA DEPENDENT KEYS 

Consider, rather, a data dependent key. For each record 
of data in the database, the key entered by the operator 
could be used to select certain bytes of the plaintext data 
which will not be protected by encryption in the record, 
which could then be used to derive the key for the encryp¬ 
tion of the data to be protected. In this way, if the key used 
to encrypt one record was discovered, that specific key 
would be useless in decrypting any other records. 

It is assumed, of course, that the data dependency of the 
key is kept as confidential as the key itself. However, if 
such is not the case, the data is still fairly well protected. 
Given that the eight bytes of the key are selected directly 
from the plaintext data with replacement, it would take on 
the average (A/*)/2 attempts to break the key where N is the 
total number of plaintext unprotected bytes available. If N 
were one hundred bytes and each attempt took one micro¬ 
second, the data would be protected for an average of over 
one hundred and fifty years. It should also be noted that the 
one microsecond figure is unreasonable for the technology 
available. The best figures to be found are those reflecting 
the implementation of the algorithm using T 2 L devices. Here 
the time to encrypt sixty-four bits is five microseconds. 7 

Protection using the above scheme could be further im¬ 
proved by not using the plaintext data itself as the key but 
rather as input to a function which creates some function of 
a key from the data chosen. A simple arithmetic maneuver, 
such as a hashing code, could increase protection manyfold 
while not significantly affecting processing overhead. Dif¬ 
ferent functions could be used for different fields and the 
types of functions used could be varied. Thus one function 
might have the property of creating widely scattered keys 
from varying plaintext. Such a key function would have a 
high velocity. Another function might be a highly nonin- 
vertible function (not one-to-one) such as adding the low 
order bits of the plaintext bytes. 


Data dependent keys improve security against the second 
method mentioned, discovering the key by statistical meth¬ 
ods. With each data record encrypted with a different key, 
the amount of encrypted data in each record would be too 
minimal to draw any conclusions about the key. 

MASSIVE DATA ENTRY SYSTEMS—ENCRYPTING 

THE SOURCE 

For a large database where data is entered by many dif¬ 
ferent people but only retrieved by a few, another scheme 
might be in order. The premise here is that some data is 
available to all for statistical purposes but only a few could 
examine or identify a specific individual. Most massive data 
entry systems employ the key to disc concept. Many oper¬ 
ators enter records at many terminals which tie into one 
processor storing the data on a disc. If this processor is 
physically local to the terminals, i.e., there is no security 
problem due to lines which can be tapped, then the encryp¬ 
tion can be done at the processor and not at the terminal. 

Given this configuration, the data can be encrypted at the 
key to disc processor using some subset of the identifying 
fields of the record, such as the name and/or the social 
security number, as the encryption key. With this key the 
protected records including the identifying fields can be en¬ 
crypted. To retrieve the data, a special decryption terminal 
can be designed which, when given the identifying infor¬ 
mation used to encrypt the record, can determine the key 
in the same manner and retrieve the record. Here one would 
extract the plain text seed for various key functions. Encrypt 
the fields that are to be protected and then encrypt the 
plaintext header information (name, social security) using as 
a key a function of extracted subfields of the plaintext header 
data. 

In this manner, only an individual authorized to use a 
decryption terminal could enter the specific name and social 
security number and from this extract the records on that 
individual. No one else could ever identify uniquely those 
individuals who had characteristics of interest. The data 
base is essentially not invertible and yet certain fields which 
are not encrypted could be made available to everyone for 
analysis. 

This scheme is superior to those previously discussed in 
that there is no plaintext stored in the database. Further¬ 
more, each encrypted record is encrypted with its own key, 
making attempts to intrude on the database via statistical 
methods futile. And perhaps most important, the data entry 
personnel do not see the encryption and may not be aware 
that it exists and do not have any retrieval capability at their 
terminal at all. 


HARDWARE IMPLEMENTED KEYS 

Protecting the key from the third source of disclosure, 
disloyal personnel, is probably the more difficult problem to 
solve. Personnel, such as terminal operators who are readily 
exposed to keys in use as well as some information con¬ 
cerning the implementation of the encryption are the weak- 
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est link in the security system. Depending on the terminal 
used, information concerning whether the key entered at the 
terminal is used to derive a data dependent key or applied 
directly as the encryption key might easily be obtainable 
through the keyboard. What is needed here then is a scheme 
by which the operator may utilize the encryption/decryption 
facilities of the terminal but without ever allowing disclosure 
of the key or the details of the implementation. 

One such scheme would involve entering the key at the 
terminal using hardware. It is possible to fashion an intelli¬ 
gent terminal with insertible ROM’s (Read Only Memories) 
similar to those used on Hewlett Packard’s 9820 calculators. 
HP 9820 users can purchase various pre-programmed 
ROM’s which when inserted in the calculator give the user 
access to a library of subroutines. 8 A terminal with a micro¬ 
processor could be so equipped and utilize subroutines on 
the ROM for encrypting, decrypting, deriving keys from 
data and supplying the main key itself to these routines on 
the ROM. What is important to note about the architecture 
of the HP 9820 is that the information on the ROM can only 
be used as instructions for the processor. The code in the 
ROM is not available to the user as data. Data and user 
supplied programs can only be entered and retrieved from 
the calculator’s RAM (Random Access Memory). It is in the 
RAM where the program which calls the ROM subroutines 
resides. 

Given a terminal with the mentioned specifications, the 
key and the method of implementing the encryption can now 
be physically protected. The ROM could easily be designed 
so that it can be locked into the terminal for safety only to 
be removed by some authorized personnel. All that the 
operator will see is the data being processed at the terminal. 

It might be useful to consider designing the terminal with 
two ROM ports for use when converting an entire database 
from encrypted under one key to encrypted under a new 
key. A ROM in one port would be used to re-encrypt the 
data. 

Security, of course, would be required when the ROM 
program is burned in. Once the implementation of the al¬ 
gorithm is established, a number of ROMs could be prepared 
at one time leaving the key field blank to be entered when 
needed and at a proper time and place. There should also 
be some method of sealing the ROM. 

Another variation of this scheme would make use of re¬ 
cently developed semiconductor devices, such as the Mo¬ 
torola “Info-guard,” 9 to perform the NBS encryption/de¬ 
cryption of the data. If this device were already present in 
the terminal and its capability available to the data process¬ 
ing program in the RAM, the ROM need only contain the 
routines for deriving and/or supplying the key to the devices. 


Using a dedicated device rather than a general purpose pro¬ 
cessor to perform encryption/decryption generally implies 
that one could expect much faster processing time. 


CONCLUSION 

The Selective Encryption Terminal is by far one of the most 
promising developments for Privacy Protection. The data is 
secure from the moment it leaves the terminal. By simply 
supplying extra protection to prevent disclosure of the en¬ 
cryption key using either data dependent keys and/or a hard¬ 
ware implemented key, a data base is well protected from 
any accidental disclosure or unfriendly intrusion. The pro¬ 
tection of data is now reduced to the physical protection of 
a device. 

Using data dependent keys for encryption and supplying 
a key to the terminal via ROM instead of through the key¬ 
board may seem awkward in terms of processing overhead 
and generating the ROMs themselves. However, each con¬ 
tributes significantly in spoiling a possible intruder’s plans 
to a point where any attempt would be completely discour¬ 
aged. These ideas, then, appear to be well worth developing. 
They may provide an attractive solution to the simultaneous 
needs of privacy and statistical reporting in government 
applications. 
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INTRODUCTION 

It may seem anomolous that electronic mail and other com¬ 
puter communication systems require cryptographic protec¬ 
tion when almost no physical mail is given this protection. 
The difference is that computer readable traffic is extremely 
vulnerable to automatic sorting at very low cost. Physical 
mail would also need to be encrypted if it were all written 
on postcards and could be sorted at a cost of only $1 for 
several million pages. Even seemingly innocuous facts can 
be damaging when such vast amounts of data can be 
screened for all messages mentioning one of a list of key 
words (e.g., computer communications, electronic mail, 
EFT, etc.). Analog voice circuits are as vulnerable to wire¬ 
tapping, but are expensive to sort. 

Fortunately, the digital nature of the data makes high 
grade encryption possible at low cost. Analog circuits are 
almost impossible to adequately secure without going 
through a digital interface and encryption. The National 
Bureau of Standards has promulgated a national data en¬ 
cryption standard which can be implemented on a single 
LSI chip. 1 In large quantities it should therefore cost on the 
order of $10, an insignificant addition to the cost of a com¬ 
puter terminal. While some have criticized the standard as 
being inadequately secure, 2-4 this is not due to technical 
constraints, but rather appears to be a political problem. 

While the cost of the encryption hardware is not a barrier 
to the widespread use of cryptography in computer oriented 
systems, there are other costs and problems which must be 
considered. Key distribution is one such problem. 5 In a 
network with n users there are approximately n 2 /2 possible 
pairs of users who may wish to converse securely from all 
other users. The distribution of this many keys by courier, 
registered mail, etc. is clearly uneconomic even for n equal 
to one million. This problem can be solved by having the 
system itself distribute keys, encrypted in user specific sys¬ 
tem keys or passwords, but this requires the system to be 
secure. 6,7 A more useful approach was suggested by Diffie 
and Heilman 5 and Merkle. 8 They proposed that it is possible 
to converse securely over an insecure channel with no prear¬ 
rangement through use of “public key systems.” The second 
section describes the public key systems of References 5 
and 8 as well as systems devised by Rivest, Shamir and 
Adleman, 9 McEliece, 10 and Merkle and Heilman. 11 


While none of these public key systems has been broken 
thus far, it is necessary that they withstand the test of time 
and concerted mock attacks by dedicated “opponents” be¬ 
fore they can be trusted because no methods are currently 
known for proving even conventional cryptosystems secure. 
In this regard we applaud the work of Simmons and Norris 12 
which looked for potential weaknesses in one of the public 
key systems. More such work is needed, and it would not 
be surprising if weaknesses were found in one or more of 
the currently known systems, much as conventional cryp¬ 
tography also went through a learning period. 

Digital signatures are discussed in the third section. Con¬ 
ventional cryptographic systems can prevent third party for¬ 
geries, but cannot settle disputes between the transmitter 
and receiver (e.g., a stock broker and his client) as to what 
message, if any, was sent. Solutions to this problem were 
first hypothesized by Diffie and Heilman 5 and found by 
Rivest, Shamir and Adleman 9 and Merkle and Heilman. 11 

PUBLIC KEY SYSTEMS 

Merkle’s public key system 8 is perhaps the simplest and 
least likely to yield to continued cryptanalytic efforts. Its 
disadvantage is that it is the most expensive. It depends on 
the existence of a one way function, a function that is easy 
to compute for all arguments in its domain but computation¬ 
ally infeasible to invert for almost all images in its range. 
Such functions have been discussed elsewhere 13-15 and are 
as easy to develop as secure cryptosystems of the conven¬ 
tional, as opposed to public key, type. 5 Merkle goes a step 
further and describes a method for generating one way func¬ 
tions of controllable “one wayness” or difficulty of inver¬ 
sion. 

A mildly one way function can be used to generate a 
puzzle, a problem which is difficult, but not impossible, to 
solve and for which it is easy to check a supposed solution. 
User A generates m keys, k v k 2 , ... , k m , and operates on 
them with a mildly one way function /(*) to obtain their 
images or puzzles, y lt y 2 , . . . , y m . These images are trans¬ 
mitted to user B who selects one of them at random, say y i7 
and solves the associated puzzle to obtain lq. The two users 
now share a key in common but A does not know which one 
it is. B therefore sends A the value z=g(/q), where g(*) is a 
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true one way function. A then operates on k x , k 2 , etc. with 
g(*) until he finds one which yields z. The successful key 
must be kq provided g(*) is one-to-one. 

The cost to A is linearly proportional to m , the number of 
puzzles, because A must operate on m k' s with/(*)—which 
is easy—to obtain the m y’s; he must store the m k' s; and 
finally must operate on approximately m/2 of the k's with 
g(*)—again an easy task—before finding z as an image. A’s 
transmission cost is also linearly proportional to m since he 
must send m y’s. B' s dominant cost is in solving the one 
puzzle that he chose at random. By making the cost of 
solving a single puzzle proportional to m, the total cost to 
the two legitimate users is still only linear in m. 

An eavesdropper, however, must solve m/2 puzzles on 
the average before finding k x , so his cost is proportional to 
m 2 . Thus as m tends to infinity the ratio of cryptanalytic 
cost to key exchange cost also tends to infinity. 

This method’s weakness is in the relatively small ratio of 
costs (m 2 :m) and the fact that the key exchange cost is as 
much in transmission as in computation. (Transmission costs 
have not decreased as rapidly as computation costs.) The 
introduction of low cost, high bandwidth transmission 
media, such as fiber optics, may make this method more 
competitive. Merkle’s paper 8 describes several very clever 
additions to this simple description, but the basic cost ratio 
is not changed. 

Diffie and Heilman 5 propose a method of public key ex¬ 
change which requires 2 bl2 operations for cryptanalysis 
(using the best known algorithm) but only b 3 operations for 
key exchange, where b is the number of bits in the repre¬ 
sentation of the key. By choosing Z>=400, key exchange 
requires only 64 million gate operations and takes approxi¬ 
mately one second in a special purpose LSI implementation. 
Cryptanalysis using currently known techniques requires 
approximately 10 60 operations and words of memory and is 
therefore totally infeasible. This technique makes use of the 
apparent one wayness of the discrete exponential function 
y=a x mod q, where q is a large prime number of appropriate 
form 16 + 7 and a is a fixed primitive element of the finite 
field GF(q). Calculating y from x—with tacit knowledge of 
a gnd q —is relatively easy, and requires only three words 
of memory, each b bits long, and b 3 gate delays. Computing 
x from y is believed tc/be much harder, and the best known 
algorithm 16,17 requires memory and time proportional to 
2 m . 

The two users and the cryptanalyst are assumed to know 
q and a. Each user generates a random number uniformly 
distributed between 2 and q-2. Call these values x, and x 2 . 
The users keep these values secret, but compute yi = a xl 
mod q and y 2 = a x2 mod q and exchange these values. The 
cryptanalyst therefore also learns y x and y 2 , but cannot fea¬ 
sibly compute x x or x 2 therefrom. User 1 takes y 2 (which 
was sent to him) and x 4 (which he has kept secret) and 
computes (y 2 ) Xl = (a X2 ) Xl = a UlX2) mod q. User 2 computes 
(yi) X2 = (« Xl ) X2 — a UlX2 \ Both users are now in possession of 
a common number K=a (x l x 2 ) mod q which they use as the 
key in a normal cryptographic system. The cryptanalyst 
cannot compute K as any obvious function ofyi andy 2 (e.g., 
y x y 2 or (yi) y 2) without first computing eitherx 4 orx 2 , which 


is an infeasible task using the best known algorithms. There 
may be better algorithms for computing x, and x 2 , or there 
may be some nonobvious method for computing K from 
andy 2 directly. As with all cryptographic systems, this one 
should be studied further to increase our trust in it. 

Merkle and Heilman 11 have proposed a public key method 
based on trap door knapsacks. Given a one-dimensional 
knapsack of length 5 and a set of n rods of lengths a t , , 

• • • , a n , one version of the knapsack problem is to find a 
subset of the rods whose lengths sum to exactly S. Equiv¬ 
alently, find a binary n-vectorx such that a*x=5. (The dot 
product of two vectors is denoted by *.) 

The knapsack problem is believed to be very difficult in 
general, and this belief is supported by its being an NP- 
complete problem. 18 In a loose sense the NP-complete prob¬ 
lems are the most difficult problems of a cryptographic na¬ 
ture. 5 A trap door knapsack vector a is one which has no 
apparent structure which can be used to simplify the solution 
process, but which possesses hidden (trap door) structure 
which allows rapid solution for x. 

As a small demonstration example, consider a=(5457, 1663, 
216, 6013, 7439) and 5 = 1663 + 6013 + 7439=15115, corre¬ 
sponding to x=(0,l,0,l,l). It happens that if each component 
of a is multiplied by 3950 mod 8443 (the secret, trap door 
information) the vector a'=(171, 196, 457, 1191, 2410) re¬ 
sults. This vector has the property that each component is 
larger than the sum of all the preceding components. Trans¬ 
forming 5 in a similar manner (multiplying it by 3950 mod 
8443) yields S'- 3797. Some thought shows that the solution 
to the problem 5=a*x is the same as the solution to 5' =a'*x, 
and that the solution to S'— a'*x is easily found because of 
the form of a'. x 5 must be 1 because S'^cu ,'—if x 5 were 0 then 
even if all other components of x were l’s the sum could not 
be large enough to yield 5'. Subtracting a 5 ' from S' yields 
3797—2410=1387 which is the sum of a subset of the re¬ 
maining components of a'. Because 1387sra 4 '=1191 we know 
that x 4 must also equal 1. Subtracting a 4 ' from 1387 yields 
196. This is smaller than a 3 ' =457 so x 3 must equal 0. It is equal 
to « 2 ' = 196 so x 2 must equal 1 and x 4 must equal 0. The 
determination of x is now complete and, as a quick check 
will show, correct for the original problem 5=a*x as well. 

Of course the trap door knapsack vector a was not gen¬ 
erated first. Rather a' was first chosen with the property 
that each component was larger than the sum of all preceding 
components and then transformed into the a vector by mul¬ 
tiplying each component of a' by 2550 mod 8443 (2550 and 
3950 are multiplicative inverses mod 8443). In a similar man¬ 
ner a program could easily be written to generate rather 
large trap door knapsack vectors from a random bit string. 
Any user of a computer system could then generate his own 
personal trap door knapsack vector regardless of his math¬ 
ematical abilities. The program would also generate the se¬ 
cret multiplier and modulus which reduces the apparently 
difficult knapsack problem 5=a*x to the trivial problem 
5'=a'*x. This program is assumed to be public knowledge 
but, even so, there is no apparent way for a cryptanalyst to 
easily solve for x only from knowledge of 5 and the public 
vector a. 

After generating a trap door knapsack vector a, the user 
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can place it in a public file. Then anyone who wishes to 
send him information can do so by representing it in binary 
blocks of n bits each, and using these as x vectors to com¬ 
pute the sums 5'=a*x which are sent to the first user, who 
can easily recover the information x even though no one 
else can. 

Note that this system is different from either of the first 
two public key systems in that a normal cryptographic sys¬ 
tem is not needed. This is because the first two systems 
each generated a number that the two legitimate parties to 
the conversation could easily compute, but neither of the 
parties could determine that number on his own. In the trap 
door knapsack system x is determined entirely by one of the 
users. While it is not necessary for x to be used as the key 
in a normal cryptographic system, in practice, the speed 
advantages of conventional cryptographic systems will prob¬ 
ably cause x to be used in that manner. This same remark 
applies to all of the currently known public key systems. 

The public key system due to Rivest, Shamir, and Adle- 
man 9 can be regarded as a generalization of a conventional 
cryptographic system developed by Pohlig and Heilman. 16 
Each user generates a pair of numbers E and n which are 
placed in a public file and which are used by others to 
encipher data they wish to send him. At the same time that 
E and n are generated, another number D is generated which 
is required for deciphering data. Clearly, it must be com¬ 
putationally infeasible to compute D from the public infor¬ 
mation E and n if the system is to be secure. As shown in 
Reference 9, computing D from E and n is equivalent to 
factoring n, and it is possible to choose n so that this is 
infeasible using the best known factoring algorithms. 

First two large prime numbers p and q are chosen and 
multiplied to produce n=pq. Then Euler’s function 
m=phi(n)=(p-l)(q- 1) is computed. Phi(n) is the number of 
integers between 1 and n which are relatively prime to n, 
and has the interesting property that almost any number 
between 1 and n when raised to the m power mod n equals 
1 (the exceptions turn out not to affect the system and we 
therefore neglect them in what follows. 9 ). E is then chosen 
as a random number between 1 and m which is relatively 
prime to m, and D is computed using Euclid’s algorithm to 
be the multiplicative inverse of E mod m . That is ED=km +1 
for some integer k. Enciphering requires only one exponen¬ 
tiation in modulo n arithmetic and is easily accomplished. 
Letting P denote the plaintext and C the ciphertext, C=P E 
mod n. (The plaintext must be represented as a sequence of 
integers each between 0 and n— 1.) Deciphering is also easily 
accomplished in one exponentiation, P=C D mod n. To see 
that this really does undo the enciphering operation note that 
C D =(P E ) D =P ED =(P m ) k P 1 =Pmodn, because = 

The most recently developed public key system is due to 
McEliece, 10 and is based on algebraic coding theory. Goppa 
codes are highly efficient error correcting codes, both in 
their error correcting capacity and in the computation re¬ 
quired to correct errors. The ease of error correction is 
destroyed, however, if the bits which make up a codeword 
are permuted prior to transmission. In McEliece’s system 
a user’s public enciphering key describes a scrambled Goppa 
code, chosen at random from a large set of possible codes. 


Anyone can easily encode information (scrambling the 
Goppa code does not greatly affect the ease of encoding 
because the code is still linear), add a randomly generated 
error vector and transmit this to the user. But only the 
intended recipient knows the inverse permutation which al¬ 
lows the errors to be corrected easily. McEliece estimates 
that a block length of 1000 bits, with 500 information bits, 
should foil cryptanalysis using the best currently known 
attacks. The main problem is the storage of a 500 by 1000 
bit generator matrix, requiring 500 kilobits of memory per 
user. 


SIGNATURES 

Written signatures are essential to our current methods of 
conducting business. They serve to indicate accountability 
and agreement on contracts, etc. Before electronic means 
can fully replace physical (hardcopy) forms of information, 
a digital equivalent to a written signature is needed. A digital 
authenticator must be a number which is easily recognized 
without being known, because any number that is known 
can be forged by the intended recipient. While at first ap¬ 
pearing to be a logical impossibility, digital signatures can 
be obtained from public key cryptosystems and probably in 
many other ways as well. 5 

Rivest, Shamir, and Adleman’s system 9 yields signatures 
most directly, merely by interchanging the enciphering and 
deciphering operations, so we only describe that method. 
When a user wishes to send a signed message M to someone 
else, he operates on it with his secret key D to obtain Y=M D 
mod n. The recipient can recover M through use of the 
public key E,n because Y E mod n=M. The recipient saves 
Y as proof that message M was sent to him by the user 
whose public key is E,n. If the sender later disclaims having 
sent the message, the recipient gives Y to a “judge” who 
can access the public file and see that Y E mod n does in fact 
equal a meaningful message with the right header informa¬ 
tion. Only the user who placed E,n in the public file knows 
D and could produce such a Y. 

In practice each block of the message will probably not 
be signed in this manner. Rather, to speed things up, the 
message will be sent in its untransformed state, and a one 
way hash total H of the message computed. 5 The signature 
will be Y=H D mod n. The recipient can easily check that H 
results when the public key E,n acts on Y, and that it is the 
same H as obtained from action of the hash function H on 
the message. 

The above discussion neglects the privacy problem which 
results if an eavesdropper may be listening. This problem is 
easily overcome by enciphering the message-signature com¬ 
bination in a normal or public key system. 

CONCLUSIONS 

Public key systems and digital signatures make teleprocess¬ 
ing systems vastly more useful for business and personal 
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use, but care must be exercised, both at the technical and 
legal levels, to ensure that these advances are not used in 
a detrimental manner. For example, a user’s secret key will 
probably be stored on a magnetic card which is needed to 
transact any business on the system. If the system becomes 
all pervasive in daily life, people may be expected to carry 
their cards with them constantly. It is only a small step to 
allow the police to demand the card as a form of universal 
identifier, without which a person becomes a nonperson. 
There are clearly dangers in such a system and adequate 
safeguards must be built in. Even now, certain businesses 
(e.g., car rental, gas stations at night) will accept only credit 
cards. 

Further research is obviously needed at a technical level. 
The security levels of the currently known systems need 
better evaluation and new systems should be sought. These 
may be more efficient than the currently known systems, or 
needed in the unlikely event that holes are found in all of 
them. A major research goal is the establishment of provably 
secure systems, conventional, public key, and signature. 
That goal is more ambitious than solving one of the premier 
outstanding problems in computer science (the P=? NP 
problem) and must be viewed as long term. 
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Legislation 


OVERVIEW 

Legislative and regulatory activity affecting the computer industry is on the 
increase. A recent computer search of legislation pending before the Congress 
retrieved 328 bills involving computers and/or communications. These bills and 
others pending before the fifty state legislatures supplement the numbers of 
existing laws that already affect the industry. Recent legislative and related 
regulatory activity has been in five general areas: communications including 
transborder data flow, privacy, protection of proprietary rights in software, per¬ 
sonnel and security. (One specific computer application, electronic funds transfer, 
which is the subject of a related area, includes aspects of each of the above.) 
Lawyers and computer specialists will describe and analyze this activity and its 
impact on computing in the following sessions: 

The session on communications entitled “Federal Policy And The Future Of 
Computer Communications Services” will focus on the present status of federal 
policy, regulation and legislation impinging on computer communications, as well 
as relevant anticipated developments and their impact on the data processing 
field. Discussion will include activity in the executive branch concerning inter¬ 
national communications and the office of the Assistant Secretary of Commerce 
for Communications and Information. Considerations of the Federal Communi¬ 
cations Commission in the ongoing FCC “Computer Inquiry” (Docket No. 20828) 
will be discussed and Congressional issues, particularly the possible revision of 
the Communications Act of 1934 and related legislation will be reviewed. 

In a session entitled “Privacy: Practical Implications For Your Organization,” 
a leader in the Federal Executive Branch Privacy Task Force will report on 
questions raised by that group’s recent review of the privacy issues, an analysis 
of the technological implications of privacy laws, regulations and recommenda¬ 
tions for organizations and projections of future development. Panelists will 
discuss the report from the viewpoints of technologists, individuals and business 
organizations. 
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The legal aspects of software protection will be discussed in a panel session in 
which the current status of patent, tradesecret and copyright law will be described 
by experts in the area—one of whom is also a Commissioner of the Federal * 
Commission On New Technological Uses Of Copyrighted Works (CONTU). An 
industry executive will then comment on the kind of protection the software 
industry needs and inquire of the other panelists how current or proposed law 
responds to these needs. 

Personnel issues in computing involve a variety of concerns. 

Computer Crime: Sources, case studies and prevention will be the topic of the 
session in which three papers will be presented that describe and defend teaching 
EDP to prison inmates, detail a recent computer crime prosecution and suggest 
a new prevention strategy based on the differences between accidental and 
intentionally caused losses. 




Software house in the big house 

by WILLARD V. HANDLEY 

Federal Prison Industries, Inc. 

Winchester, Virginia 


INTRODUCTION 

At the United States Penitentiary, Leavenworth, Kansas, 
and the Federal Correctional Institution, Milan, Michigan, 
some fifty convicted felons are learning to be computer 
programmers. Inmates supplement academic training with 
experience by developing systems for the Veterans Admin¬ 
istration, Department of Labor, and other Federal agencies. 
Programming services are sold to agencies at market prices 
to generate revenue for inmate salaries and other expenses. 
This custom software enterprise is, or course, managed and 
supervised by civil service staff. 

A software house whose programmer staff are convicted 
felons must pay attention to the potential of computer fraud 
in its security program. But it is not enough to have a 
program deemed adequate by experts in the field. To the 
general public, the computer is still an awesome and mys¬ 
terious device. Our operation, and indeed any computer 
operation susceptible to public opinion must, like Caesar’s 
wife, not only be above reproach but appear above reproach. 

We learned this fundamental lesson when an allegation 
was made that inmates were using our computers to defraud 
IRS. Nothing of the sort had happened, nor could happen, 
under the stringent procedures in force at Leavenworth. We 
knew it; but we could not convince our critics, because the 
explanation was too complex. 

This paper explains how we subsequently changed our 
security program to make it more clearly obvious to any 
observer that inmates could not misuse our resources. It 
should prove useful not only to other prison systems with 
similar rehabilitation programs, but to any organization af¬ 
fected by public opinion. Everyone may not have inmates 
as programmers, but most have some form of Achilles’ heel. 
The need to go beyond “secure” to “obviously secure” in 
your own sensitive area of security may be as great as was 
ours. 

The IRS episode also resulted in criticism of the funda¬ 
mental purpose of our program. There are some who argue 
that a skill so easily turned against society should not be 
taught to known felons. Although an inmate, once released 
from custody, is outside the scope of our security program, 
this paper would not be complete without an explanation of 
why we feel the program is a boon, rather than a bane, to 
society. 


COMPUTER PROGRAMMING IN FPI 

The Federal Prison System houses some 30,000 offenders 
in prisons located throughout the country. The Prison Sys¬ 
tem has a duty beyond merely keeping them isolated from 
us. It is expected to afford inmates humane treatment and 
the opportunity for rehabilitation. 

Humane treatment includes helping the inmate pass the 
endless hours without losing his sanity. This entails the need 
for meaningful work programs; ideally, these programs con¬ 
tribute also to rehabilitation by teaching a skill usable on the 
“outside.” 

“Rehabilitating offenders” is a phrase not now heard in 
the field of corrections as often as before. Now we admit 
the best we can do is provide opportunities for self-improve¬ 
ment. So, education, vocational training, religious oppor¬ 
tunities, and work programs are made available, on a vol¬ 
untary basis, along with close counseling by social workers. 
Some inmates sincerely avail themselves of these opportu¬ 
nities; others merely “do their time,” and often it is these 
who come back to prison again and again. 

The Leavenworth programming enterprise is only one of 
many self-supporting work/training programs operated by 
Federal Prison Industries, Incorporated. This wholly-owned 
government corporation was established by Congress in 
1934 to operate as a private manufacturing concern, buying 
raw materials, producing products, and selling them to gen¬ 
erate its operating revenue. Its management and supervisors 
are government civil servants; its workers, inmates. It was 
given freedom to develop and manufacture any product, but 
restricted to selling to other Federal agencies. It was capi¬ 
talized with $4 million of machinery, equipment, and other 
assets with no provision for subsequent Federal funding. 

Today, FPI, now operating under the trade name of “UNI- 
COR,” has seventy factories and shops in thirty-two penal 
institutions throughout the country. It is virtually a conglom¬ 
erate, with six product line divisions and forty industrial 
activities. UNICOR makes textiles; several lines of office 
furniture; footwear; wiring harnesses for missiles, tanks and 
aircraft; operates print plants, sign factories, microfilming 
shops engineering drafting shops, and other graphic arts 
facilities; and offers data entry, systems analysis, and com¬ 
puter programming, and other commodities and services. It 
employs 5,900 inmates, 600 civilians, and has annual sales 
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of over $90,000,000. It has been self-sustaining for the forty- 
three years of its existence. Unlike some government cor¬ 
porations, and not a few private ones, it has never needed 
another infusion of Federal funds. 

Probably the most ambitious UNICOR effort to provide 
inmates usable skills was the establishment, in 1972, of the 
Leavenworth computer programming industry. As with 
most UNICOR industries, its workers have little or no prior 
experience. A year of classroom training, taught by certified 
teachers, is provided to prepare carefully selected inmates 
for the program. Graduates then work in a closely super¬ 
vised environment to increase their skills with live work. 

Students are selected from among a usually long list of 
volunteer applicants who have been recommended by their 
counselors. The potential trainee must have a high school 
diploma or GED certificate, perform at the tenth grade level 
in general math and language arts, and score a minimum of 
45 out of a possible 90 points on the IBM Programmer 
Aptitude Test. A panel of custody, counselor, and industry 
personnel reviews such documents as the inmate’s prior 
criminal record, pre-sentencing report, caseworker’s prog¬ 
ress reports, and overall incarceration history to reach a 
decision on each applicant’s suitability for the program. 

The course consists of thirteen four-week instructional 
modules designed to provide the equivalent of vocational 
school programmer training. It includes hands-on training 
with an NCR Century 200 computer with 64k bytes of main 
memory, dual disk storage units, a card reader, and line 
printer. The system software includes the operating system, 
assembler, and both FORTRAN and COBOL compilers. 
The course stresses COBOL, although an introduction to 
the assembler language and to FORTRAN are included. One 
module introduces IBM JCL and operating system utilities.* 

A new course begins roughly every six months to start a 
new class of fifteen students on the road to becoming pro¬ 
grammers. By the end of the year, dropouts and elimination 
by rigorous testing has dwindeled the group to about 12. 
This number is adequate to maintain production staffing at 
about thirty programmers and eight analysts. 

Production programmers are grouped into six or seven 
man teams, carefully staffed to maintain a balance of ex¬ 
perienced and novice programmers. A non-working civilian 
supervisor, qualified by years of experience, heads each 
team. The leader is responsible not only for the quality and 
quantity of each team’s production, but for each member’s 
continued development and growth. 

Additional staff required to operate the facility includes 
two Leavenworth-based analysts and two Washington-based 
analysts/marketing representatives. The Leavenworth ana¬ 
lysts are augmented by a team of four inmate analysts each. 
An administrative officer and a computer room operator 
round out the staff. 

The training program currently costs over $150,000 an¬ 
nually; operating costs for the production element bring the 
total to nearly $550,000. But with the potential for over 
60,000 programmer hours per year, the industry is still ca- 


* A copy of the curriculum is provided at the end of this paper. 


pable of operating “in the black’’ despite the unusual need 
to extensively train every programmer. 

When the program began in 1972, a customer was found 
conveniently close to Leavenworth: The Agriculture and 
Stabilization and Conservation Service in Kansas City. The 
ASCS sent Leavenworth detailed programming specifica¬ 
tions which were reviewed and clarified by Leavenworth 
analysts before assignment to the teams. Leavenworth was 
soon turning out programs whose reject rate of 8.7 percent 
was deemed by the customer “much better than average.” 
Inmates were honing their skills on live production work, 
and were beginning to find jobs in data processing upon 
release. By June 1976, twenty-four of the forty-eight re¬ 
leased had become programmers or analysts in private in¬ 
dustry. 

It soon became apparent, however, that having one cus¬ 
tomer was not enough. To provide a continuous flow of 
work and keep all programmers busy most of the time, we 
needed several customers to fill in the gaps. Turning to other 
agencies, however, meant computer compatibility problems, 
since the Federal government, by design, has many different 
computers. Because of our proximity to ASCS/Kansas City, 
we had been able to use the customer’s machine remotely; 
now we were looking at the distant market in Washington, 
D.C., where such arrangements might be impractical. 

The obvious solution was commercial time-sharing. For 
each job, we would find a vendor who offered the same 
machine as our potential customer’s, and an existing net¬ 
work to give us local access. 

We started with Infonet and its UNIVAC 1108s. Inmates 
quickly learned to use time-sharing and the new tools it 
afforded, such as interactive programming. Infonet had not 
been chosen casually; it boasted more Federal cus¬ 
tomers than any other network. By contracting with these 
agencies, we could develop systems on the same computer 
as our customers thus eliminating compatibility problems. 

That on-line access to computers shared with Internal 
Revenue, the Veterans Administration and others also posed 
the hazard of subversion was not overlooked. The initial, 
and as it turned out, only, terminal was placed in a secure 
room. Civilians made all telephone contacts, and observed 
inmates continuously as they used the machine. Plans for 
additional terminals included a PBX to be modified by the 
telephone company so that outgoing calls could be made 
only at the switchboard, in response to verbal requests by 
civilian staff. The switchboard would be located in the ad¬ 
ministrative area of the prison, outside the main security 
gates. All telephone lines would be disconnected at the 
switchboard at 4:00 P.M. 

But we had not gone far beyond the training stage with 
this new tool before an unrelated incident occurred in an¬ 
other part of the prison that embroiled us in a national 
controversy, and eventually eliminated time-sharing as a 
resource of this program. 

A CRISIS AND ITS IMPACT 

Among Leavenworth’s population at that time was an 
inmate busily defrauding the Internal Revenue Service. 
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There was nothing sophisticated in his operation; he was 
taking advantage of the commonly known fact that IRS does 
not crosscheck, except on a limited scale, the amount a 
taxpayer’s W-2 form shows was withheld with what his 
employer reports. To his phony tax returns showing money 
he had not earned, the inmate simply attached bogus W-2 
forms showing taxes that had never been withheld. IRS 
processed the returns routinely, and sent a refund check to 
his wife on the outside. 

The inmate was eventually caught and prosecuted. Al¬ 
though he was not connected with our program in any way, 
it was perhaps inevitable that it would be assumed our com¬ 
puter was being used in this scheme. Overnight, the story 
spread and grew by leaps and bounds. In major newspapers, 
wire service reports, and national television news broad¬ 
casts, it was implied, and sometimes stated outright, that 
inmates trained by the prison were using computers to crack 
the IRS security codes, file hundreds of bogus returns, and 
embezzle millions of dollars. 

Computer security experts’ statements, such as “No com¬ 
puter system exists that cannot be penetrated” made it dif¬ 
ficult for the press to understand our assertions that our 
inmates could not penetrate Infonet. No one wanted to hear 
that, as a practical matter, penetrating a system requires 
certain resources, such as a prior knowledge of the operating 
system and unlimited access to a terminal, neither of which 
our staff had. In the end, we suffered as much adverse 
publicity as had we been guilty. 

FPI Security 

The Leavenworth security program had to counter three 
distinct possibilities for computer fraud at the time of the 
IRS scandal: 

1. Client System Penetration. An inmate might penetrate 
the Infonet system of controls and gain unauthorized 
access to sensitive data of the FPI client or other In¬ 
fonet users. He might also modify copies of programs 
stored on Infonet so that they would produce some¬ 
thing of value to himself at some future time. Likewise, 
he might penetrate the ASCS IBM 360 machine to 
which the Leavenworth computer was linked. 

2. Client System Exploitation. An inmate might produce 
a program that, in addition to its intended function, 
secretly provided some benefit, such as issuing a 
check to the programmer, or indirectly benefited him 
by alerting the client’s operating system to subvert his 
security controls for later unauthorized use. 

3. Unauthorized Use of Computer Resources. An inmate 
might use the Leavenworth computer or the Infonet 
system to directly produce something of value for him¬ 
self, such as an official-appearing transcript; or to run 
a lottery. 

These same threats exist in most data processing organi¬ 
zations. 1 We felt at the time that, while we should do every¬ 
thing possible to reduce the probability that these threats 


would be carried out, we would be doing nothing more than 
any conscientious software house should be doing. 

Our countermeasures at that time were based on controls 
that eliminated the personal freedom and access to equip¬ 
ment needed to carry out the threats previously described. 
A large measure of control was provided as a byproduct of 
being in a prison setting, and of having a staff comprised of 
trainees and “green” programmers. 

An important factor in all publicized computer crimes to 
date, and indeed, in virtually all white collar crime, is the 
numerous privileges and freedoms enjoyed by the perpetra¬ 
tors. 2,4 But the incarcerated Leavenworth inmate leads a life 
of almost constant supervision. He is deprived of the usual 
freedoms, not out of fear he will commit a computer crime, 
but because his custody is the responsibility of some prison 
employee at all times. He must have express permission to 
move from one area of the institution to the other, and must 
always be in a prescribed place at a prescribed time. At 
night, he is locked away in a cellblock separated from the 
computer room by numerous barred doors and gates. 

During working hours, he has the freedom to be in the 
computer programming area, but still the requirements of 
custody apply. His supervisor must be constantly aware that 
he is present and doing his assigned job. No “outside” 
programmers are so closely observed. 

Few would contest the statement that in the usual pro¬ 
gramming environment, managers are not enough in touch 
with their programmers to constantly know what they are 
doing. Thus a programmer runs little risk of discovery in 
appearing to be doing legitimate work while actually utilizing 
the computer for fraudulent purposes. 

A close relationship between team leaders and program¬ 
mer is assured at Leavenworth because the primary mission 
is different from that of the public sector software house. 
The supreme goal here is to build each inmate into an 
accomplished programmer so that he can succeed in the 
field upon release, rather than to make money for the en¬ 
terprise. The team leader works closely with the inmate 
from the initial stages of specification interpretation, through 
flowcharting, debugging, and testing. This is a well-recog¬ 
nized concept for promoting program integrity, which in 
turn helps to ensure that a program does only what it is 
intended to do, and nothing more. 1,3 

Thus, the environment in which this software house op¬ 
erates provides a degree of control not usually found in the 
public sector. But additional, more direct security measures 
were also in force at the time of the IRS incident to counter 
the three main threats: 


1. Management Control of Computer Runs —All run 
decks are submitted to the inmate’s team leader, who 
reviews each deck to ensure that the job is one assigned 
to the inmate, and that the JCL meets prescribed stand¬ 
ards: Nothing is suppressed on the output listings, no 
permanent files are created, and only the test files 
prepared by civilian staff are accessed. Decks may be 
brought into the locked computer room only by the 
team leader, who also must retrieve the printed output. 
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Every listing is reviewed by the team leader before 
being turned over to the programmer. 

2. Limiting the Possibilities —Live test data is never in¬ 
troduced into the prison, nor were inmates given access 
to live data when shared computers were being used. 
An unwritten policy at the time of the incident, since 
formalized, was to avoid accepting systems to be pro¬ 
grammed that provided an opportunity for financial 
gain. 

3. Personnel Selection —The screening process previ¬ 
ously described for inmates being considered for entry 
into the program was designed not only to ensure that 
the man had the ability for the program, but, to the 
extent possible, that he had a sincere desire to change 
his life for the better. Admittedly, this judgment is 
subjective. 

4. Independent Testing and Review of Programs —ASCS 
policy required that all programs be compiled and re¬ 
tested with their own test data against the program 
specifications and flowcharts. ASCS also ran each pro¬ 
gram through the Performance Monitor of METACO- 
BOL, a product that helps in determining program in¬ 
tegrity by listing unexecuted paragraphs. 

Most professionals would agree that the above proce¬ 
dures, if consistly and conscientiously applied, would come 
as close to eliminating the possibility of computer fraud by 
Leavenworth inmates as possible, given the existence of the 
threats outlined. Indeed, a detailed analysis of this environ¬ 
ment by the MITRE Corporation, well-known for its efforts 
in security on behalf of the Department of Defense, led them 
to state in their first report following the IRS incident: “The 
security procedures in use at this time are probably stronger 
than those used in most software development shops, and 
are more than adequate for the types of programs currently 
being developed and the current development system.” 4 

Nevertheless, we realized that this was not enough to 
prevent a repetition of the incident and its consequence of 
adverse publicity. An indirect threat existed for which we 
had no countermeasure: The public outcry against a danger 
more perceived than real. It could spell the end of our 
program. A combination of a general belief that with com¬ 
puters, anything is possible, and the prejudice that everyone 
who has previously committed a crime will do it again, given 
an opportunity, meant the issue of computer fraud was ex¬ 
tremely sensitive in our situation. 

The conclusion was that countermeasures against the 
three threats was not enough; we had to go further and 
eliminate as many of the threats themselves as possible. 
Under the guidance of the MITRE Corporation, we moved 
to reduce all threats to the issue of program integrity, and 
then to maximize our countermeasures against it. 

This is being accomplished by isolating the development 
process from the client or target system. We withdrew from 
Infonet and will sever the connection with the ASCS ma¬ 
chine, and begin using our own dedicated machine for sys¬ 
tem development. Thus, the only thing that wiH flow be¬ 
tween the development facility and the client system is the 
program itself, which will be moved manually. The ‘dedi¬ 


cated facility’ is a well-known security technique, recog¬ 
nized by Department of Defense. 5 

The residual issue of ‘program integrity’ was addressed 
by strengthening the procedures already in effect. The tech¬ 
nique of structured programming was adopted to help ex¬ 
pose faulty or malicious code. A comprehensive, but com¬ 
mon-sense, set of procedures and controls was also adopted 
to protect the program from the time it is “cleared” by the 
Leavenworth staff until installed on the target system. To 
further reduce the threat of program integrity, we have de¬ 
fined a set of formal guidelines against which to screen 
future client systems to reduce as much as possible the 
chance that an inmate will work on a program that could 
benefit him. These exclude: Programs that directly update 
data files that are used as the basis of financial transactions; 
programs in any way involved in the disbursement or trans¬ 
fer of funds to individuals, organizations, or accounts; and 
programs that access files which contain classified or ex¬ 
tremely sensitive data that could be used for immediate or 
future financial gain. 

All of these procedures are designed to protect society 
from the inmate programmer still incarcerated. They of 
course have no effect of the inmate programmer once he is 
released. The IRS fraud scandal raised another issue that 
still appears from time to time: Is it proper for the Federal 
government to turn known criminals into computer program¬ 
mers? It is interesting to note that in a report following its 
investigation of the Leavenworth incident, a Congressional 
committee criticized the use of timesharing and advised 
against FPI involvement in systems that disburse funds, but 
approved of the program itself. 6 

TEACHING PROGRAMMING IN PRISONS 

The basic argument against teaching computer program¬ 
ming is that it provides criminals with tools to commit more 
rewarding crimes, with less effort and risk than their pre¬ 
vious adventures required. If the saying “once a criminal, 
always a criminal” were true, this argument would be un¬ 
assailable. But it isn’t; nearly two-thirds of all Federal re¬ 
leasees have been shown to be “successes.”* 

Because we are dealing with something so complex and 
unpredictable as human beings, it is impossible to say just 
why some releasees succeed in remaining out of prison while 
others fail. But we have clues. We know that people who 
commit crimes have the same basic desires as the rest of 
us; security, status, and so forth. The main difference seems 
to be that the criminal is more impatient, wanting his desires 
met immediately. And they often lack the training to fulfill 
their desires through legitimate means. 

The FPI data processing program is a long-term, rigorous 
ordeal. An inmate must apply himself diligently for twelve 
months of training to avoid being dropped from the course. 
After completing the course, he has to put in two or three 


* From “Success and Failure of Federal Offenders Released in 1970,” A 
Bureau of Prisons study which defines a “success” as having no parole 
revocation or new sentence of 60 days or more. 
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years of further training in live production work, steadily 
improving his skills, in order to land a job upon release. 
Many never make it through the first few months of training. 

Earlier release is not the motivating factor for these par¬ 
ticipants. The Parole Board gives no consideration to the 
programs an inmate completes. But, over the years, inmates 
have watched our participants released to well-paying, in¬ 
teresting jobs, and they hear through correspondence about 
the satisfaction these people get out of their new lives. This 
is the predominant reason given for wanting to participate 
in our program. 

The program itself, then, is self-screening. We feel it at¬ 
tracts and keeps only the inmate who has made a firm 
commitment to play by the rules. The attitude of wanting 
instant, effortless success would not carry a trainee halfway 
through the first course. It is difficult to imagine someone 
still caught up in the syndrome of instant gratification having 
the patience to undergo such a grueling ordeal. 

We feel the program is worthwhile because it provides an 
opportunity for those suited for a data processing career, 
who otherwise might continue robbing banks, or stealing 
securities to change their behavior. We have clear indica¬ 
tions, after six years of operation, that our successful par¬ 
ticipants do go on to lead productive lives. 

While it is difficult under existing procedures to follow up 
individuals released from custody, we can tell from the 
FBI’s National Crime Information Center Computerized 
Criminal History File whether a releasee commits a new 
offense. From a continual review of these records, we 
know that no former participant has ever been charged with 
a computer crime. Moreover, these records indicate that 85 
percent of our successful participants have had no further 
involvement of any kind with the law. 

Even sophisticated statistical studies (which the above is 
not) dealing with the issue of relating a releasee’s rehabili¬ 
tation to his custodial treatment are subject to attack. There 
are simply too many factors that account for human behav¬ 
ior. Because the data, as positive as it appears, is inconclu¬ 
sive, one has to look elsewhere for additional indications 
that the society is better served by continuing the program. 
One such indicator is the attitude of private firms who hire 
our “graduates.” 

I hesitate to list these firms for fear of possibily embar¬ 
rassing ex-inmates now leading productive lives in their em¬ 
ployment. Key officials in these companies are of course 
aware of the men’s backgrounds, but peers in some cases 
may not be. These companies would be recognized imme¬ 
diately by name. They are national and international in scope 
and have become prominent through hardnosed decision¬ 
making. These firms have decided that hiring ex-inmates as 
computer professionals is good business, and send the same 
teams that conduct campus interviews to our “campus.” 
Recurrent hiring by the same firm is not uncommon. 

It seems, in the final analysis, that judging whether teach¬ 
ing inmates to program computers is wise is, and should be, 
in the hands of these companies that hire our releasees. It 
is they, after all, who stand to lose if our critics are right. 
If they felt that the program was dangerous, they could 
stop its effects, and the program itself, by simply refusing 


to hire participants. Instead, they urge us to continue; one 
firm tells us they get more from our experienced people than 
from fresh college graduates. 

Computer programming in the Federal institutions will 
therefore go on. Our internal security procedures, and the 
attitudes of the inmate participants, are adequate to prevent 
inmates’ misusing of this knowledge while in our custody. 
Because private sector employers have confidence in the 
program and its graduates, the data processing community 
as a whole should accept these individuals, not as criminals 
with new tools, but former offenders making a useful con¬ 
tribution to society. 
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APPENDIX —CURRICUL UM 

The one-year training phase is designed around behavior/ 
skill producing modules which permit: 

Flexibility in teaching and learning in a penitentiary; 
Frequently measured goals which culminate in the PTU 
objectives, and, 

Accurately defined functional areas of computer program¬ 
ming technology. 

Thirteen modules are established as follows (each four 
weeks long, 160 classroom hours) with the objectives as 
stated. 

Computer Programming Voctational Training by Behavioral 
Module 

1. The Computer and its Program, People, and Periph¬ 
erals. An introduction to computer programming as 
a technical science. 

2. The Scientific Programmer. Learning and applying 
the FORTRAN programming language. 

3. The Business Programmer. Learning and applying the 
COBOL programming language up through the use of 
tables and index-sequential files. 

4. The Systems Programmer. Learning and applying the 
NEAT/3, Level 1 programming language. 

5. Operating a Computer for Fun and Profit. Computer 
operations. 

6. The Computer Utilities and Maintenance Men. The 
use of software utilities to perform discrete jobs, and 
the modification of existing computer programs. 
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7. All De-Buggers are not Household Insect Sprays. The 
art of writing a computer program from specification 
to completed program, without waste or futility. 

8. The Super Business Programmer. Advanced COBOL 
through internal sorts, conversion to IBM from NCR, 
and beginning JCL. 

9. NCR versus IBM—How Does it work? IBM utilities 
and advanced JCL and relationships to NCR’s ways. 

10. Garbage In, Garbage Out. Designing data systems, 
and test data. 

11. Critical Issues Seminar in Computer Science. Discus¬ 
sion of current developments in the field, i.e., com¬ 
puter security, information privacy, etc. Use of out¬ 


side guest lecturers such as NCR/IBM representatives, 
prior releasees, representatives from users organiza¬ 
tions. 

12. 1 think you heard what I meant, but I’m not sure I 
said what you think you heard! Documenting a com¬ 
puter program and current documentation standards 
required in the production phase. 

13. Where do we go from here? Student evaluation. This 
module broken into four parts for use by instructors 
to evaluate students at 3, 6, and 9 month intervals, 
and at the end of the course of instruction for ac¬ 
ceptance in the production phase. 
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INTRODUCTION 

Research papers generally treat the problem of computer 
security as an intentional act to be solved by intellectually 
challenging methods such as encryption, descriptor-based 
systems, and program proving. Accidental and intentional 
acts are almost always referred to simultaneously; however, 
the safeguards are designed strictly to prevent intentional 
acts. It is generally thought that the same safeguards and 
strategies will be effective against both accidental and inten¬ 
tional acts. This is not necessarily true. 

In a paper on security assessment by Glaseman, Turn and 
Gaines, 1 accidental and intentional differences are not stated 
but are made subservient to the question of exploitation of 
vulnerabilities. As with most research papers on security, 
important research issues are discussed but they are oriented 
principally to intentional acts. This is evident in use of terms 
such as “intruder,” “attacks,” and “exploit,” but no dis¬ 
cussion penetrates research needs for errors, omissions and 
other accidents. 

In contrast, Courtney states in a paper on risk assess¬ 
ment: 2 

It is important that proper weight be given to the impact 
of errors and omissions. Data is more often destroyed 
or otherwise rendered useless or even harmful by people 
making mistakes than through dishonesty or malice. 
People whose loyalty and honesty are unquestioned but 
whose judgment and competence leave much to be de¬ 
sired are our greatest enemies. Data security consid¬ 
eration must not be limited to concern for the acts of 
dishonest people. Otherwise, it is very difficult to 
achieve proper cost justification of appropriate security 
measures. Weigh all potential security problems. . . 
The differences between accidentally and intentionally 
caused losses are many and profound. 

Unfortunately, he does not pursue these strong position 
statements to exploit the differences. 


* The preparation of this paper was supported, in part, by the National 
Science Foundation Grant #MCS76-01242. However, any views or conclu¬ 
sions in this paper should not be interpreted as representing the official 
position or policy of the sponsor or SRI International. 


The differences between accidental and intentional acts 
are profound. Accordingly, the requirements for the strate¬ 
gies and implementation of computer safeguards should re¬ 
spond specifically to each kind of act. Nevertheless, most 
research stops short of considering two security problems. 
It is therefore suggested that a more successful approach to 
the problem would be to regard the accidental and inten¬ 
tional acts as distinct entities requiring separate solutions. 
Treating each act separately would make the problem-solv¬ 
ing process more manageable and more explicit so that com¬ 
bining the two derived strategies and safeguard applications 
will result in a more effective approach to both aspects of 
computer security. 

The first step is to identify characteristics of the two 
problems. Each of the characteristics is discussed below. 


CHARACTERISTICS OF ACCIDENTAL AND 

INTENTIONAL ACTS 

Frequency 

Incidence of errors and omissions is usually sufficiently 
high to support statistical analysis of loss cases in a com¬ 
puter service organization. A limited number of cases, how¬ 
ever, is often tolerated because the expense of reducing 
incidence any further is not cost-effective. Analysis can be 
used to identify the major locations and sources of errors 
and omissions. Statistics on the incidence of accidents can 
be used to calculate the probability and expected loss from 
these problems. 

Conversely, the low incidence of intentionally caused 
losses makes statistical analysis difficult. Only in limited 
cases, such as credit card and some other payment frauds, 
has the incidence been high enough to be subject to statis¬ 
tical analysis. 3 Most intentional acts, especially those in¬ 
volving significant losses, occur so infrequently that at¬ 
tempts to calculate probability, expected loss, nature, 
patterns of location, and source, are difficult if not impos¬ 
sible. Therefore, risk analysis at the present time must be 
more subjective and theoretical. The experience of victims 
that could be similar to experiences of future or potential 
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victims may be helpful in calculating statistical probabilities, 
but will probably not be sufficient. 

Unsolved problems 

There are few unsolved problems with regard to acciden¬ 
tally caused losses. Efforts to control unintentional losses 
have been made throughout the development of electronic 
data processing. Adequate solutions are known for most 
common errors and omissions. The causes of errors and 
omissions are mostly obv jus and the solutions equally ob¬ 
vious. It remains to apply rhese solutions in a cost-effective 
manner. Nonetheless, despite the errors, computer opera¬ 
ting systems, for example, are still effective. 

In contrast, significant problems in security against inten¬ 
tionally caused losses remain unsolved. The technical se¬ 
curity of currently available computer systems is inadequate 
against attacks from potential perpetrators (such as systems 
programmers) with sufficient skills, knowledge, and access 
to the systems. Moreover, computer operating systems are 
basically unpredictable. Therefore, proving the integrity of 
a system and guaranteeing freedom from unauthorized 
changes or enhancements are not possible at this time. Nei¬ 
ther practical prevention nor adequate detection capabilities 
are currently known for such sophisticated attacks as Trojan 
Horse methods, salami techniques, data leakage, logic 
bombs, asynchronous attacks, or post system failure at¬ 
tacks. 4 Another problem yet to be solved is the adequate 
identification verification of legitimate remote system users. 
Where technical safeguards are inadequate, therefore, more 
difficult and costly methods of safeguarding must be devel¬ 
oped for protection from intentional acts. 

Act complexity 

Accidentally caused losses derive from single, isolated 
acts. Each loss incident is usually independent of other loss 
incidents. On the other hand, an intentionally caused loss 
can result from sequences of dependent and independent 
authorized and unauthorized acts. The loss can occur from 
intentional efforts that capitalize on observed errors and 
omissions. Perpetrators often increase the complexity of 
intentional acts as a means of avoiding detection or appre¬ 
hension. 

Singularity of source 

One person usually is responsible for an error or omission 
even though other people may cause extenuations, whereas 
more than one individual frequently perpetrates an inten¬ 
tional act. Half the known cases of computer abuse have 
involved collusion. Compared to manual, or non-computer 
related fraud, embezzlement, etc., there is much more col¬ 
lusion in perpetrating these crimes via computer. The reason 
for the high degree of collusion is that computer crime re¬ 
quires more skills, knowledge, access, time or resources 


than any one person usually possesses in the technically 
oriented environments of the computer systems. Security 
strategies and safeguards must be far more elaborate when 
one attempts to deal with the possibility of collusion. 

Complexity of perpetrator behavior 

Behavior of people causing errors and omissions is rela¬ 
tively simple. The behavior is related to the conditions at 
the instant of the act. After the act, the perpetrator need, at 
most, defend only his weakness that resulted in error. In 
contrast, the behavior of people performing intentional acts 
is often highly complex. Interviews with 23 people who 
intentionally perpetrated a computer crime revealed that 
these people had personal problems to solve or goals to 
reach preceding their search for vulnerabilities of a computer 
system or work environment. 3 Searching or studying a sys¬ 
tem for vulnerabilities is, however, only one facet of the 
complex behavior pattern of intentional perpetrators. Once 
motivated to penetrate and use a system to his own ends, 
a perpetrator plans, plots, gathers information, organizes, 
and conspires, and is left finally to rationalize all of his 
intentional acts. Effective security must address all of these 
issues. 


Sources of security assistance 

Accidentally caused losses have been extensively studied 
and reported in the technical literature for many years. As 
stated previously, however, most technical literature that 
claims to address both accidental and intentional acts effec¬ 
tively addresses only accidentally caused losses. Profes¬ 
sional societies and governmental organizations dedicated 
to the prevention of intentionally caused losses have only 
recently attempted to examine the problems associated with 
computer technology. The American Society for Industrial 
Security, the U.S. Justice Department, the International 
Association of Chiefs of Police, National Crime Prevention 
Association, and Surety Association of America have 
barely, if at all, seriously or significantly addressed the crim¬ 
inal aspects of computer usage. 

Security checklists 

Much security literature relies heavily on checklist or 
cookbook approaches to computer security. 5,6 Strategy is 
based On the concept of implementing the well-known safe¬ 
guards and controls identified in the checklists. Although 
this approach is particularly effective for handling errors and 
omissions, it is neither sufficient nor effective in dealing 
with intentionally caused losses. Perpetrators are aware of 
the safeguards and controls identified in checklists, thus 
their methods for compromising the system are designed to 
avoid them. Merely installing a named safeguard in a check¬ 
list is not necessarily sufficient protection from intentional 
acts. Moreover, the checklist rarely includes information 
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concerning the need and means of protecting the safeguard 
itself from attack or compromise. 

Strategy and safeguard independence 

Each error or omission possibility can be effectively 
treated in isolated ways, because one loss will be independ¬ 
ent from any other loss. Thus, a safeguard against an acci¬ 
dental loss can be implemented without considering other 
possible losses. Basically, the formula of matching one error 
with one safeguard is reliable. 

On the other hand, each intentionally caused loss possi¬ 
bility must be covered comprehensively, including all pos¬ 
sible acts leading to the loss. A safeguard against an inten¬ 
tional act will often have impact on other types of acts or 
losses in that it may only partially supply adequate protec¬ 
tion. A particular loss could result in a security strategy 
involving many safeguards or a defensive depth of multiple 
rings of safeguards which may be a desirable strategy. Fur¬ 
ther, the implications of imposing a safeguard must be con¬ 
sidered in terms of its potential to open new possibilities of 
vulnerabilities. If a particular safeguard temporarily deters 
a perpetrator, he will simply seek ways around it or other 
vulnerabilities. 

Safeguard compromise 

In accidental loss situations, it can generally be assumed 
that the integrity of the safeguard will be ensured and sus¬ 
tained. Safeguards may be subject to failures from errors, 
but in general the errors will be independent of the errors 
that the safeguard is preventing. In contrast, safeguards are 
subject to intentional attack as part of the perpetrator’s 
strategy in carrying out an intentional act. The compromise 
of safeguards becomes part of the total loss event that the 
safeguards are meant to prevent or detect. In this sense, a 
safeguard must be considered as an asset subject to protec¬ 
tion if it is to perform adequately when needed. 

Level of protection 

Determining the level of protection that a safeguard from 
an accidental act affords is simple because of the one-for- 
one, act-safeguard relationship. Higher incidence of acci¬ 
dents can result in more easily calculated protection levels. 
In intentionally caused acts, the level of protection is deter¬ 
mined by a combination of security measures associated 
with a given asset. Lack of statistics makes effectiveness 
difficult to determine. 


Achievable protection level 

Security from accidental losses can be achieved to a high 
degree or, at least, can be easily controlled at a wide range 
of loss levels. One reason for this is that the success of 


protection can be accurately measured from incidence sta¬ 
tistics, and the security then can be changed on the basis of 
the need that is determined. Because there are few unsolved 
protection problems with accidental losses, it is possible to 
control incidents to any degree desired within the limits of 
economic considerations. 

A high level of protection from intentionally caused losses 
is difficult to achieve because of incomplete knowledge of 
all vulnerabilities. An individual loss could be affected by 
many factors, including the difficult-to-determine complete¬ 
ness of protection. The one-upmanship escalation of pro¬ 
tection and increasing sophistication of perpetrators’ meth¬ 
ods precludes reliable measurement. 

Potential perpetrators 

The population of potential perpetrators of accidental loss 
is easily identified. It consists of those persons having the 
access and authorization to perform an act that might result 
in an accidental loss. Conversely, the population of potential 
perpetrators of intentionally caused losses is difficult to 
identify. It includes not only those identified relative to 
accidentally caused losses, but also the often larger numbers 
of people who might gain the necessary skills, knowledge, 
and access to perform an unauthorized act. For example, in 
remote terminal usage, impersonation alone would account 
for a large number of people who might possess such skills 
to replace people in positions of trust. 

Potential perpetrator capabilities 

In accidentally caused losses, the minimum skills, knowl¬ 
edge, and resources of potential perpetrators are at issue. 
Access is a prerequisite. Beyond this, the security specialist 
is dealing with the minimum skills and knowledge of people 
that are the source of errors and omissions. With intention¬ 
ally caused losses, the security specialist is dealing with 
people who potentially have the maximum possible skills, 
knowledge, access authorization, resources, and time to 
commit the act. The attack possibilities and matching pro¬ 
tection activities constitute a game in which each side is 
pitting its maximum capabilities against the other side. 

Loss limits 

Accidentally caused losses are limited in size. Where the 
size of the loss can vary, the probability of detection and 
termination of the loss grows rapidly with the size of the 
loss. The victim is ordinarily able to observe, measure, and 
limit accidental losses in a timely manner. However, detec¬ 
tion of intentionally caused losses in a timely manner is not 
necessarily related to the size of the loss. Exceptionally 
large losses can go undetected when the perpetrator puts 
forth significant effort to conceal the losses. Also, in increas¬ 
ingly automated criminal methods, the possibility of a large 
loss in a time scale measured in milliseconds before humans 
can react becomes more likely. 
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Detection 

Perpetrators of accidentally caused losses have no con¬ 
scious intention to err before and during their acts. There¬ 
fore, avoidance of detection occurs, if at all, after the act 
has occurred. There is generally less fear of detection or of 
reporting the loss and more cooperation with the victim in 
recovery in accidental loss situations as compared to the 
perpetrators’ position relative to intentional acts. Interviews 
have revealed that perpetrators greatly fear unanticipated 
detection before, during, and after their acts, and often much 
of their efforts and resources go into prevention of detection. 
This makes detection of intentionally caused losses a much 
greater challenge than for accidentally caused losses and 
should occupy a greater amount of the security specialist’s 
attention. 

Theoretical approach 

Support for claiming a difference in risk between acciden¬ 
tal and intentional acts can be shown in mathematical terms, 
even though this procedure is highly theoretical and not 
intended for practical use. Given are an asset subject to 
accidentally or intentionally caused loss, its environment, 
and a population of potential perpetrators with known skills, 
knowledge, access, resources, time and motivation. The 
probability P of a particular type and size of loss L from any 
possible independent accidental acts a t (/= 1,2,. . . n) with 
probabilities of their occurrence p(a t ) is P(L) = 'h n p(a i ) - T\ n p 
(oj). This suggests that a safeguard that reduces the proba¬ 
bility of an act a t for any value of i will reduce P(L). 

The probability P of a particular type and size of loss L 
fromanypossibleindependentintentionalacts«i(/=l,2,. . . n) 
with probabilities of their occurrence p{a t ) is P(L)=Ma.x n {p(a i )}. 
This conclusion is based on the theoretical premise that any 
potential perpetrator will perform rationally in choosing the 
one act with the greatest probability of success. This sug¬ 
gests that an entirely different result would occur from in¬ 
stalling a safeguard that reduces the probability of an inten¬ 
tional act tf; for any i. If it were the act with greatest 
probability, it would have the most significant effect on P(L) 
by directly reducing it by the amount that would cause the 
act of next largest probability to determine the value of 
P{L). If it were any other act that was affected, there would 
be no change in P{L). Therefore, installing safeguards has 
no effect except for those that reduce the probability of the 
act with greatest probability of occurrence. 

Two different security strategies are suggested by these 
theoretical conclusions. Optimum strategy for security from 
accidentally caused losses is to reduce the probabilities of 
that combination of acts that reduce P(L ) by the greatest 
amount to an acceptable level, given limited security re¬ 
sources. Optimum strategy for security from intentional acts 
that reduce P(L) to an acceptable level is to reduce the 
probability of the act with maximum probability followed in 
sequence by reducing the probability of subsequently max¬ 
imum probability acts until security resources have been 


expended or P(L ) is reduced to an acceptable level, which¬ 
ever occurs first. 

Applying any safeguard that reduces the probability of 
any accidental act will incrementally increase security. 
However, applying a safeguard that reduces the probability 
of an intentional act may not increase security unless it 
reduces the probability of the one act that represents the 
maximum probability of occurrence. Thus, security from 
intentional acts is no greater than the weakest link. This 
shows a major difference in the two kinds of problems. 

The most obvious weakness in this argument is the as¬ 
sumption that the population of potential perpetrators is 
rational in always selecting the act with the greatest possi¬ 
bility of success. In addition, practical application of the 
theory implies that the population and all acts relative to 
loss of a particular asset are known. 

Applying a strategy only for accident prevention or only 
for intentional act prevention would clearly result in subop¬ 
timization, because many safeguards, some with minor mod¬ 
ification, serve both purposes. A combined strategy, al¬ 
though not fully optimized, offers significant advantages 
over the near random-based, currently used strategies that 
are organized around merely computer center functional 
approaches (physical, operational, procedural and system) 
to security strategy. The steps to be followed in applying a 
combined strategy are: 

(1) Select the intentional act with largest probability of 
occurrence. 

(2) Apply safeguards to reduce the probability. 

(3) Identify the accidental acts also reduced in probability 
of occurrence. 

(4) Repeat Steps (1), (2), and (3) with the newly identified 
intentional act with largest probability until probability 
of intentionally caused loss is reduced to an acceptable 
level. 

(5) If the probability of accidentally caused loss is still 
not acceptably lowered, apply safeguards to reduce 
the probabilities of the combination of acts that reduce 
the probability of accidental loss to an acceptable 
level. 

The alternatives of applying safeguards against accidental 
loss first or applying safeguards without regard to accidental 
or intentional act differences are less effective. Safeguards 
aimed only at accidental acts often will be ineffective or will 
not be the best safeguards against intentional acts. One 
reason for this is that the safeguards may be installed only 
with the intent of protecting against the compromise of as¬ 
sets and not with the intent of protecting the safeguards 
themselves against compromise. In addition, a particular 
order of implementation of safeguards may not be respon¬ 
sive to the strategies of the potential perpetrators who are 
looking for the simplest and safest route to achieving suc¬ 
cess. 

Failure to use the recommended strategy results in the 
Maginot Line Syndrome: A monolith of obvious and tradi¬ 
tional safeguards will be installed, and the perpetrator 
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merely bypasses it by finding the weakest link not yet ad¬ 
dressed by the victim organization. 

/ 

/ 

ILLUSTRATION 

! ! 

Implementation of access control by password to an on¬ 
line system from a terminal illustrates the problem. For 
purely accidental access prevention in a benign environ^ 
ment, identification verification by input of a minimal length 
password or name is all that is necessary. Prevention of 
intentional, unauthorized access to the system requires the 
following additional measures: 

• Sufficient password length to reduce exhaustive search 
attempts. 

• Assignment of randomly generated passwords to reduce 
guessing by analysis of dependence between password 
content and password holder characteristics or knowl¬ 
edge. 

• Nonvisible input of passwords to avoid observation or 
scavenging. 

• Frequent briefings of password holders concerning 
safekeeping of passwords. 

• Safe password administration, including separation of 
duties; invocation of need-to-know principle; appropri¬ 
ate transport and storage of passwords; and specific 
authorization of duties to accountable and trusted em¬ 
ployees to avoid spoofing, coercion, degradation of 
care, and unauthorized physical access attempts. 

• Encryption of on-line system password files and pass¬ 
word comparison in encrypted form in privileged mode 
computer operation to prevent technical compromise. 

• Limit guessing attempts of unauthorized passwords by 
imposing time delays between input attempts and dis¬ 
connect after n failed attempts. 

• Record and verify password use periodically with pass¬ 
word users to avoid password theft and unauthorized 
use. 

• Analyze patterns of password use and failed access 
attempts for deviations from normal experience to de¬ 
tect unanticipated attacks. 

• Test all protection mechanisms frequently and prepare 
and follow contingency plans to avoid attacks when 
system or operational failures occur. 

• Perform frequent, random, independent, and visible au¬ 
dits to inhibit potential perpetrators and ensure safe¬ 
guard integrity. 

• Impose sanctions against violators. 


These additional safeguards are not new, but it is difficult 
to find them all stated in a single source. Again, subsets of 
them'have been implemented, but are not generally found 
in a single computer service organization. For prevention of 
accidental access, only a few are needed. For prevention of 
intentional acts, all are needed; however, total implemen¬ 
tation of all safeguards still is insufficient protection in many 
high-risk environments. Many more detailed specifications 
are needed for such items as levels of encryption effective¬ 
ness, pattern analysis of password use, password length, 
and audit techniques. 

CONCLUSION 

Glaseman, Turn and Gaines conclude by stating two major 
requirements for progress in security assessment: 

(1) Increased research aimed at the development of a 
better understanding of the informational elements of 
security assessment, and 

(2) Experience at the level of individual computer instal¬ 
lations, in the application of a broader and more ac¬ 
curate information base to the assessment of computer 
security. 

A third requirement should be appended to those suggested 
above: 

Recognition and separate treatment of two separate 
and distinct computer security problems, accidentally 
caused losses and intentionally caused losses. 
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Anatomy of a computer crime* 


by SUSAN HUBBELL NYCUM 

Chickering & Gregory 
San Francisco, California 


INTRODUCTION 

This paper describes the process of an investigation** into 
the technical and legal aspects of a computer abuse of al¬ 
leged theft of programs stored in a computer. The goals of 
such investigation are to provide information to help make 
computers and computer systems more secure, to provide 
computer technologists and managers with data as to the 
potential exposure for a particular activity, and to provide 
law enforcement officers and prosecutors knowledge of 
whether and to what extent existing laws may apply to the 
case. 

There is no wish on the part of the investigators to sen¬ 
sationalize a case or bring to the victims and perpetrators 
more notoriety or exposure than already exists from public 
documents and proceedings or prior press coverage. There¬ 
fore, the paper will not use actual names and will where 
possible, generalize rather than particularize places and or¬ 
ganizations. 

Methodology of investigation 

To date the investigators have knowledge of over 500 
reported cases of computer abuse. To investigate each of 
these is beyond the existing level of resources. The cases 
that are selected for field study are those which offer one or 
more of the following: 

(1) a plethora of available detail; 

(2) an interesting, yet typical rather than wildly bizarre 
modus operandi; 

(3) a suspected or demonstrated significant gain to the 
perpetrator or similar loss or exposure to the victim; 

(4) a nontrivial legal question as to what crime or legal 
wrong has been committed; and 

(5) a nontrivial investigative question for the law enforce¬ 
ment agency. 


* The preparation of this paper was supported, in part, by the National 
Science Foundation Grant #MCS76-01242. However, any views or conclu¬ 
sions in this paper should not be interpreted as representing the official 
position or policy of the sponsor or Chickering & Gregory. 

** As performed by Susan Nycum and Donn Parker pursuant to the above 
National Science Foundation Grant. 


These five topics are more fully developed as follows: 

(1) The incident must be available for research. The proj¬ 
ect prefers to investigate an incident at the postresolution 
stage. This qualification more readily assures that the parties 
will candidly share their experiences. While individuals tend 
to rationalize their actions at all stages of an incident, post¬ 
conviction interviews with perpetrators are more candid 
than pretrial or presentencing interviews. Similarly prose¬ 
cutors and law enforcement personnel are understandably 
constrained from discussing a case before conclusion. Vic¬ 
tims considering or in the course of civil litigation are nor¬ 
mally uncommunicative.* 

All aspects of the incident must be open to research to 
avoid the three common barriers to objectivity: bias, prej¬ 
udice and interest of the parties thereto. In addition, each 
person involved sees the incident from a different perspec¬ 
tive. As with the old anecdote about the blindfolded group 
asked to describe an elephant by touch, one at the trunk, 
one at the tail, one at the feet, each party to an incident 
“sees” the case somewhat differently and extrapolates the 
whole from his own experience. The investigator needs to 
put those various viewpoints together and can best do so 
from a wide variety of interviews. 

(2) There are always unique approaches to the perpetra¬ 
tion of computer abuses and there are always “old friends” 
of tried and true approaches. Since the project has six years 
of experience, many of the typical approaches have previ¬ 
ously been identified, investigated and catalogued. If an 
incident looks like a good example of a known modus op¬ 
erandi, there is low utility, absent extenuating circumstan¬ 
ces, in investigating that type again. At the same time a “far 
out” approach, dependent on a set of unique circumstances 
has lower utility than an abuse which could happen repeat¬ 
edly to a larger set of victims, e.g., point of sale terminal 
users. This is not to say that new twists to old abuses or 
sophisticated efforts which thwart existing security are ig¬ 
nored. These may be the most critical cases from which to 
learn improved security techniques. 

(3) The project does not chase the illicit makers of a 


* However, many victims and perpetrators have given information to the 
project in confidence and that confidence is carefully guarded by the inves¬ 
tigators. 
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snoopy calendar* any more than a traditional investigator is 
concerned with the typical business practice of taking pen¬ 
cils from the office for home use. The project prefers to 
spend time where the results of the study will help minimize 
exposure to.nontrivial risks. 

(4) From the standpoint of legal research, abuses to soft¬ 
ware and services are of greater significance than those to 
hardware. This is because hardware, however complex, is 
visible, touchable material, the generic equivalent of other 
materials the abuses to which the law has coped with for 
centuries. Software and computer services, e.g., computer 
time, are new forms of intangible property and less amenable 
to traditional legal treatment. 

(5) How does one find out in what way the abuse hap¬ 
pened, how does one prove what happened for the purpose 
of prosecuting the act and later implementing security 
against similar occurrences and implementing proper audit 
trails. This criteria can be evaluated with respect to inven¬ 
tiveness of the modus operandi or with respect to the rules 
related to evidence gathering, e.g., telephone traces. 

Once an incident is selected for investigation, the project 
contacts the cognizant persons, the victim, perpetrator(s), 
law enforcement persons: prosecutors, and judicial person¬ 
nel and sets up personal appointments with these people as 
nearly contiguous in time as possible at the interviewee’s 
office or residence (including jail cells where applicable). 
(This helps to preserve the continuity of the investigation 
and to assure that similar questions and similar weighting 
techniques as to response are used.) 

At an interview the researcher explains the project goal 
and invites a description of the event in the interviewee’s 
own words. The approach is one of openness and profes¬ 
sional interest. There is no attempt to intimidate or lead the 
interviewee. There is also a minimum level of sympathy or 
support for action taken. Experience indicates that rapport 
can be established with the interviewees without compro¬ 
mise to the interview—the project is not on one side or the 
other. New points that arise in a later interview are checked 
for comment by earlier interviewees on a call back basis. 
There are patterns to interviews but these vary with the type 
of abuse and the class of interviewee, e.g. victim, prosecu¬ 
tor, perpetrator. Notes of interviews are transcribed as soon 
as possible after the fact and details of the investigation 
rechecked and gaps filled in and authorities explained by 
subsequent contacts. 

Results of case studies are added to an aggregate of data 
on several issues, project findings are never based on one 
incident. 

Selection of the Brown** case for investigation 

The Brown case was a theft of proprietary software from 
a private installation over telephone lines. It was investi¬ 


* Note, however, that S. 1766, The Federal Computer Systems Protection 
Act of 1977, makes such activity susceptible to a $50,000 fine or 15-year 
prison sentence. 

** An actual case with names changed. 


gated by the FBI, prosecuted by a U.S. attorney and the 
defendant pled not guilty. At a resulting jury trial the de¬ 
fendant was convicted on two counts and one count was 
dismissed. The facts were typical of a particular form of 
abuse which had not been previously investigated in depth. 
There were interesting legal and investigative issues and the 
motive of the perpetrator was in dispute at the trial. All 
parties were available for interview. 

THE FACTS OF THE BROWN CASE 

The facts of this incident have been undisputed since the 
perpetration was discovered and the perpetrator appre¬ 
hended. The main and perhaps only contention is as to why 
the act was done. 

The parties agree that the perpetration consisted of the 
unauthorized copying of a significant portion of a systems 
program from the installation on which it was running by 
means of remote terminal access over dial-up telephone 
lines. 

The perpetrator (“Brown”) was a former employee who 
had had supervisory status for technical conversion and 
operation of a data center, at the victim company. Brown 
had executed a confidential information and invention agree¬ 
ment at the time of hire which stated in relevant part: 

“I will not disclose to anyone outside of [Company] or 
use in other than [Company’s] business any confidential 
information, information or material, including but not 
limited to any and all information, material, data or mem¬ 
oranda relating directly or indirectly to any technique, 
tool or method used and/or applied by [Company] in any 
computer program or project related to the business of 
[Company] or its subsidiaries, either during or after my 
[Company] employment except with [Company’s] written 
permission.”* 

All the accessing of the system and copying of the program 
was done after Brown left the company. 

The program copied was a heavily modified variant of a 
system originally developed by an educational institution 
and financed in whole or in part by government grants to 
that educational institution. No data was copied and no 
other program was copied. 

Detection 

The perpetration was originally discovered by an alert 
employee who noticed that an imposter was signed onto the 
system. The detection of the presence of an imposter was 
possible because the person whose identification was being 
used was in sight of the alert employee and not engaged in 
a terminal session. System capabilities were used to monitor 
the terminal session and telephone company traces specified 
the origin of the terminal use. 


* Official Transcript, pp. 27-28 of Proceedings. 
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Prosecution 

As the telephone access was across state lines, the FBI 
investigated the incident and the cognizant U.S. attorney 
prosecuted the perpetrator for violation of federal law, Sec¬ 
tion 1343 of Title 18 of the United States Code (wire fraud).* ** 
The indictment was summarized by the judge as follows: 

“That the defendant, [Brown], obtained the unpublished 
telephone access number to the computer systems oper¬ 
ated at and by [Company] exclusively for [Government]; 

That the defendant unlawfully and without authority 
obtained account numbers, user identification numbers, 
and other information which was necessary in order for 
him to access the said computer and computer systems; 

That the defendant, through the use of said information, 
gained access to the computer systems operated by [Com¬ 
pany] for the exclusive use of [Government], including 
the system known as [X]; 

That the defendant without authorization used said 
identification numbers and account numbers, and without 
authorization obtained information, including print-outs, 
from the computers and computer systems, then knowing 
that he did not have authorization to gain such access and 
to retrieve such information. 

... It is further alleged in each of Count I and II that for 
the purpose of executing said scheme to defraud, the 
defendant caused to be transmitted in interstate commerce 
by means of a wire communication, namely a telephone 
communication between the defendant in [Y State] and 
[Company in Z State], certain signs, signals and sounds. 

Count I alleges a transmission on our about_, 

1975 and Count II alleges a transmission on or about 
_, 1975. 

In summary, then, the defendant is charged with devis¬ 
ing a scheme to defraud and with using a wire transmission 
in furtherance of that scheme. The same scheme is in¬ 
volved in each count, although different transmission have 
been alleged.”* 

A third count which alleged receipt of stolen property, 
was dismissed by the judge who ruled that the electronic 
impulses transmitted by telephone line across state lines 
could not constitute stolen goods because they were not 
tangible items. 

The defendant admitted the act of copying the system but 
argued first that the system was in the public domain and 
thus he hadn’t taken anything proprietary from an owner 
and second that his intention was not to use the system for 
his own interests but to show the vulnerability of the com¬ 
pany’s systems security. 


* §1343. Fraud by wire, radio, or television. Whoever, having devised or 
intending to devise any. scheme or artifice to defraud, or for obtaining money 
or property by means of false or fraudulent pretenses, representations, or 
promises, transmits or causes to be transmitted by means of wire, radio, or 
television communication in interstate or foreign commerce, any writings, 
signs, signals, pictures, or sounds for the purpose of executing such scheme 
or artifice, shall be fined not more than $1,000 or imprisoned not more than 
five years, or both. Added July 16, 1952, c. 879, § 18(a), 66 Stat. 722, and 
amended July 11, 1956, c. 561, 70 Stat. 523. 

** Transcript pp. 832-833. 


After a two and one-half hour deliberation, the jury found 
the defendant guilty on both counts. 

Similarity to other cases 

The fact pattern of this incident was close to that of an 
abuse which took place in California several years earlier. 
In the former case a programmer pled guilty to theft of trade 
secrets. These trade secrets consisted of a proprietary ap¬ 
plication program the perpetrator had copied over remote 
telephone lines to his employer’s computer and then printed 
out a hard copy thereof on his employer’s printer. The 
perpetrator in that case admitted that he intended to use the 
program for competition on behalf of his employer against 
the victim. 

The contention of the perpetrator in the Brown case that 
his intention when he copied the program was to show the 
Company that its systems security was inadequate. This is 
most reminiscent of a 1974 Oregon case of computer abuse 
in which a student notified the console operator that he had 
taken control of the computer operating system from a ter¬ 
minal and had done so for the purpose of pointing out the 
security vulnerability of the system. The system in the Or¬ 
egon case stored motor vehicle registration data including 
personal information. The system in the Brown case was 
targeted to contain government data classified “secret.” 

UTILITY OF THE CASE TO THE STUDY 

The case was of great utility to the study of computer 
abuse from several aspects. 

Security implications 

The victim company described its systems access controls 
as a three part key number access: account number, asso¬ 
ciated user identification and password all of which had to 
be provided by the user and verified by the system before 
computer time could be used and files could be accessed. In 
their interview with the study the spokesmen were candid 
about their concern for the security breach but indicated the 
level of existing security was adequate to protect from ex¬ 
ternal intrusion. Another employee of the company stated 
in a separate interview that steps had been taken after the 
incident to tighten the level of security. Testimony at trial 
indicated a security weakness in that passwords were infre¬ 
quently changed: 

Q. How often are the passwords changed? 

A. Right now, they are changing once a week. 

Q. Well, up until the time of this little problem, how often 
were they being changed? 

A. At that particular time, I couldn’t tell you, but it was 
not very often. 

Q. Well, I guess the next question obviously is, has the 
password been changed periodically as a result of this 
incident? 

A. They were supposed to be changed periodically before 
the incident. 
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Q. Well, I think that’s clear, but I mean did this incident 
bring on a policy of changing those passwords? 

A. No, the policy was there prior to that. 

Q. Did this incident bring on an implementation of the 
policy of changing passwords? 

A. Yes, it did change the password, yes. 

Q. Okay, because they found that to be a weakness in the 
system, isn’t that fair to say? 

A. That’s correct. 

Q. And how about the identification numbers now, are 
they being changed more frequently? 

A. The systems people’s identifications are being changed 
more frequently. 

Q. Are they becoming more complex? 

A. The format remains the same.* 

The prosecutor 

The prosecutor stated that this had been the first exposure 
of that office to computer abuse and they felt hampered by 
the lack of adequate federal law applicable to the case and 
by their own unfamiliarity with the type of activity. (The 
FBI could not be interviewed without a many months’ delay 
because of backlog of requests under the Freedom of Infor¬ 
mation Act. Their reports, however, had been used by the 
U.S. attorney who did share his experience on the case with 
the study.) 

Impact on programmers 

From the standpoint of programmers involved with pro¬ 
grams developed other than by themselves entirely with 
their own resources, the case points out two critical matters. 

Most programs are owned by someone; very, very few 
are actually in the public domain. Indeed, another paper by 
the study is directed to a discussion of the ownership of 
programs whose development was sponsored in whole or in 
part by government funds. If the perpetrator Brown thought 
himself innocent of any wrongdoing when he took a copy of 
the company program, he was woefully mistaken. That as¬ 
sumption, which is unfortunately widespread, is wrong. Pro¬ 
grammers must assume, unless assured otherwise by the 
persons responsible for the program, that it belongs to some¬ 
one and permission to copy or use the program must be 
obtained in advance of such use. 

System security is a legitimate concern of all those in¬ 
volved with computer installations and programs running 
thereon. If Brown was truthful, his efforts to convince the 
company during his employment of its lack of systems se¬ 
curity through memo and meetings were unsuccessful. His 
testimony was that having failed to achieve better security 
while an ei ployee of the company, Brown intended to copy 
the entire program after termination and present the com¬ 
pany with the evidence of a series of perpetrations and the 
significant results of such accesses. (In fact he accessed the 


* Official Transcript, pp. 88-89. 


system approximately 60 times without authority.)* The jury 
disbelieved Brown. His case might have been stronger had 
those noble intentions been documented or had he in any 
way enunciated his concerns to users of the facility or to 
law enforcement personnel. Computer people whose per¬ 
sonal ethics require action similar to Brown must realize 
and acknowledge that the action carries with it the possibil¬ 
ity of conviction of a crime. 

Legal aspects of the case 

The case was interesting from a legal standpoint for sev¬ 
eral reasons. 

The Judge’s dismissal of the count of receipt of stolen 
property was the second time a jurist had been faced with 
the unauthorized transfer of a copy of a computer program 
in the form of electronic impulses over telephone lines. In 
both cases the judges found this act to be outside the pur¬ 
view of the penal laws affecting asportation (carrying away) 
of personal property. 

The prosecutors felt hampered in their conduct of the case 
by the lack of directly applicable law. They were fortunate 
that the alleged theft had been interstate and thereby subject 
to federal jurisdiction and the federal wire fraud statute. Had 
the incident occurred entirely intrastate, the laws of the 
particular state involved might not have provided the source 
of a prosecutable offense. 

The character of the property involved, software, exposed 
the prosecution and the defense to the complicated and 
confusing issue of identification and protection of proprie¬ 
tary interests in software. This issue, as discussed above, 
was also a source of stated misunderstanding by Brown. 

From a security standpoint, the methods of perpetration 
and detection were of value in determining how to make 
computer systems more secure, both technically and oper¬ 
ationally. 

CONCLUSIONS 

Conclusions reached from a study of this incident include: 

1. There is a need to educate prosecutors and defense 
attorneys about computer abuse. This was not the first 
or only instance of compromise of a computer system 
and a theft of a program and it was not the only instance 
of such a compromise being made purportedly not for 
gain. It would have been helpful to each side to have 
known of other occurrences. 

2. Prosecutors and law enforcement officers need infor¬ 
mation about the environment and terminology of com¬ 
puter installations so that investigation and case prep¬ 
aration can proceed more independently of the victim’s 
guidance. 

3. Computer organizations need to know how to protect 
proprietary software, legally and physically. 


* Transcript p. 749. 
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4. Computer professionals need to be aware of their legal 
exposure for appropriation for their own purposes of 
software not owned solely by themselves. 

5. Computer organizations need to realize that security 
precautions are an ongoing concern and that proce¬ 
dures must be kept current to remain effective, partic¬ 
ularly vis-a-vis current and former employees who 
know the system. 
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Federal policy and the future of computer 
communications services 


by PHILIP S. NYBORG 

Computer and Communications Industry Association 
Arlington, Virginia 


Communications policy has long been a matter of direct 
Federal concern, based on the government’s responsibility 
to regulate the monopoly communications industry. As com¬ 
puter technology becomes increasingly reliant on commu¬ 
nications services, and as computers become increasingly 
utilized in the provision of communications services, the 
Federal government is faced with difficult policy decisions 
in executing its mandate to regulate communications ser¬ 
vices while at the same time refraining from regulation and 
inhibition of developments in computer systems. 

In the intersection of the computer and communications 
industries, there are major new markets emerging for data 
communications services as well as hardware which argua¬ 
bly has both data processing and communications functions. 
Equally important, other regulated services which have tra¬ 
ditionally been provided in paper media, such as postal 
service and banking, are now shifting to electronic form as 
electronic mail and electronic fund transfers. The techno¬ 
logical possibility of all digital networks, in which a wide 
variety of services including voice could be distributed in 
digital form under computer control, further complicates the 
problem of regulating monopoly communications carriers 
while at the same time maintaining a competitive free en¬ 
terprise market in the nonregulated data processing field. 

Incoming Federal Communications Commission (FCC) 
chairman, Charles D. Ferris, while affirming the FCC pro- 
competitive posture toward the provision of certain com¬ 
munications services has recognized the scope of the new 
technological opportunities. Ferris was recently quoted in 
the Washington Post as saying “I don’t think its Buck Rog¬ 
ers to conceive of a system in the very near future of homes 
and certainly businesses having not only voice communi¬ 


cations with each other but access to data banks, even video 
communications” and citing the possibility of computer ter¬ 
minals in each home and business for transactions ranging 
from banking to ordering groceries. 

The panel will address the present status of Federal pol¬ 
icy, regulation and legislation impinging on computer com¬ 
munications, as well as relevant anticipated developments 
and their impact on the data processing field. 

The discussion will commence with a brief overview of 
multiple regulatory jurisdictions within the Federal govern¬ 
ment which affect computer communications in their various 
forms, including the Federal Communications Commission, 
the Congress and groups within the Executive Branch. The 
panel will then focus on the major issues within these for¬ 
ums, beginning with consideration of issues currently ad¬ 
dressed by the Executive Branch; of particular importance 
will be international communications, as well as the newly 
created office of the Assistant Secretary of Commerce for 
Communications and Information. Following will be two 
closely related presentations on regulation of the commu¬ 
nications industry, as implemented by the FCC and the 
Congress. Discussion of the Federal Communications Com¬ 
mission will address the trend toward a pro-competitive 
policy and the distinction between communications and data 
processing which is being formulated in the ongoing FCC 
“Computer Inquiry” (Docket No. 20828). Current issues 
before the Congress will be discussed as an extension of 
recent FCC decisions, and particular attention will be given 
to the possible revision of the Communications Act of 1934 
as well as other related legislation dealing with industry 
structure in the telecommunications field. 
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Area Director: 

David C. Rine 

Western Illinois University 

Macomb, Illinois 


careers and education 


The Technical Area of Computing Careers and Education offers those attending 
NCC-78 a state-of-the-art view of topics including Professional Requirements and 
Educational Alternatives, Accreditation of Information Processing Programs, 
Professionalism and Professional Development, Career Paths for Women, and 
Computers in Early Education. Viewpoints will be presented as to what educa¬ 
tional skills and backgrounds recent graduates should have as they enter positions 
in industry and government. Needs will be identified for a formal evaluation 
program that will accredit the offerings of particular institutions and programs, 
similar to accreditation guidelines for computer engineering programs. Treat¬ 
ments of obsolescence problems through professional development for the com¬ 
puting professional, including guidelines for defining and conducting an individual 
professional development program, will be covered. It will, also, be postulated 
that women must more carefully design their careers in information processing. 
Finally, calculators and microcomputers are now cheap enough so that pre¬ 
schools, elementary schools and high schools can afford to give students quite 
good access to them, and better training and education with them. 
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A treatment—Professional development 


by FRED A. GLUCKSON 

National Bank of Detroit 
Detroit, Michigan 

“There’s something going on here 
but you don’t know what it is . . 

Bob Dylan 


PREFACE 

In the mid-1940’s, the most advanced technology around 
was the Hydra-Matic transmission. The conventional wis¬ 
dom among us kids was that the Hydra-Matic was so com¬ 
plex that no single person understood it all. A look at its 
valve body, which directed hydraulic pressure along various 
paths to effect gear changes, seemed to prove this. But the 
logical complexity of this valve was roughly the same as 
today’s single-bit binary adder circuit. And such a circuit is 
an inconsequential component of today’s most advanced 
technology—as exemplified by the Intel 8085 microproces¬ 
sor, which has 6,200 transistors, can execute 770,000 in¬ 
structions per second, and is housed on a chip .164 inch by 
.222 inch—smaller than Lincoln’s head on a penny! In only 
30 years the frontier of technology has surpassed anything 
we kids could have imagined. Most of us in this room will 
be alive in 30 more years. Where can the leading edge 
possibly be then? 

Next, let us consider the career of Albert M. Diederich at 
the Ridge Tool Company. Mr. Diederich assembles 
wrenches. He has been assembling wrenches for 45 years. 
In that time, Mr. Diederich estimates that he has built 25,- 
000,000 wrenches! 1 He is, by all odds, the world’s ranking 
expert at his trade; there is no problem beyond his knowl¬ 
edge or ability; no one can teach Mr. Diederich “a thing or 
two.” Can anyone in this audience claim such expertise? 
Not likely! This is the nature of a career in information 
sciences: ours is a field in which experts disagree, where 
terms are ambiguous, but in which our achievements have 
been profound. We have used computers, in the Viking 
space program, to design and build a radio antenna so sen¬ 
sitive that it can perceive the energy radiated by a lighted 
match held on the surface of Mars. 2 We have designed a 
computer program that plays world-class chess and has 
beaten David Levy, an international chess master, at the 
computer’s forte—speed chess—but not yet in regulation 
tournament chess. Mr. Levy described this program at ACM 
77 in Seattle. “Chess 4.6 is fantastic. Nine years ago I 
wouldn’t have believed it possible. It doesn’t make tactical 
oversights. You can’t fool it. But it can’t make a long-term 
strategic plan. If it could, it would be a grandmaster.” 


Meanwhile, back in retail sales, a clerk waves a wand and 
the cash register prints the item description and price while 
inventory control systems a thousand miles away are noti¬ 
fied of the transaction. In the classical sense, this is a magic 
wand. But for us, the data processors of today, the tech¬ 
nology of such an application is well within our understand¬ 
ing. Or ought to be. Unlike Albert Diederich, we have cho¬ 
sen a profession that will leave each of us hopelessly behind 
if we do not search aggressively and persistently to stay 
with our art. 


INTRODUCTION 

It is my intention to suggest a program of self-development, 
aimed both at remedying known personal deficiencies and 
at keeping abreast of new applications of computing tech¬ 
nology. There are many opportunities for self-development 
and I will point these out. But it remains for the individual 
computing professional to take the initiative to set up a 
program for himself or herself and to follow thru. It will 
require a commitment of time, and in some cases, money. 
But how could personal resources be better expended than 
as an investment in one’s future? 

One’s career may be affected profoundly, as follows: em¬ 
ployee appraisal forms usually have a section labelled “Job 
Competence”. Other sections relate to personal traits such 
as “Leadership Skills” or “Interpersonal Relations”. It is 
difficult for adults to change basic personality attributes 
such as leadership skills. So there is not a great deal that we 
can do about low ratings in non-technical areas. But we 
have no excuse for continuing to earn low ratings in tech¬ 
nical knowledge. 

When a computer professional takes a new job, his first 
performance appraisal may be lower than usual because of 
the new skill requirements. But having identified these de¬ 
ficiencies and established a program to remedy them, sub¬ 
sequent appraisals should improve. So there is little excuse 
for one’s career progress being limited by technical inade¬ 
quacy. 

One large corporation strives to channel five percent of 
everyone’s time into technical training. This comes to two 
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and one-half weeks a year, or two hours a week for formal¬ 
ized training apart from reading journals. If your organiza¬ 
tion is similarly enlightened, so much the better. If not, your 
five percent will have to come from personal time. Surely 
no one in this audience surrounds himself with TV and beer 
in free time. I suggest that you do not reduce the time spent 
with family—that is more important than career. Rather, 
you should defer some leisure time activities and substitute 
the pursuit of your own professional development goals. If 
this is too much of a burden, you may be in the wrong 
profession. The one resource each of us has in equal meas¬ 
ure is time. Other facilities are not so evenly distributed. 
Personal wealth is not, nor is intelligence or good looks. But 
each of us has exactly 24 hours each day to be allocated as 
he sees fit. A major factor in success is one’s ability to 
manage time to best advantage. The old cliche is: “If you 
need something done, ask a busy person to do it.” You 
should not schedule yourself to be constantly busy. Relax¬ 
ation is important to physical and mental health. But you 
should schedule your relaxation—not just let it happen. Do 
not drop in unexpectedly on a busy friend and expect a 
warm welcome. 

Make a list of things to do and the priority of each. The 
first item on your list should be “make a list;” then when 
you finish the list you can immediately scratch off the first 
task and you have made progress. 

OBJECTIVE 

Perhaps an overall objective for self-development would 
be to help us distinguish good ideas from marginal ones. Not 
all ideas work; not all products are successful. Remember 
Corfam? Du Pont invested a king’s ransom in this synthetic 
substitute for shoe leather. They ultimately wrote off this 
investment as a total loss. Does Viatron ring a bell? Ten 
years ago this company set about building a full-function 
terminal to rent for $21 per month. A good friend of mine, 
a computer professional, used pencil and slide rule (remem¬ 
ber those?) to show that it could not be done. Viatron ulti¬ 
mately went into receivership. It takes years for the mar¬ 
ketplace to render these judgments. A computer professional 
may be able to use intuition and analysis to render an in¬ 
formed appraisal. As of this writing the Commodore PET 
computer is described by its Project Manager 3 as selling for 
under $600, with volume projections of 5,000 per month. 
Service is to be handled by the retailer. Is this “viable”? 
What about conferencing by computer? Will this happen? 
Here is Dolotta on the subject: 

“We believe that over the next ten years the potential 

market for applications aimed at such ‘offices of the fu¬ 
ture’ may well equal the value of all of today’s business 

data processing applications.” 4 

Structured programming and distributed processing are 
current buzzwords. The computing professional ought to be 
well enough informed to have an understanding, and perhaps 
an opinion of these concepts. It is not sufficient to have 


written the world’s tightest routine for converting 1401 five¬ 
digit operand addresses to properly zoned three-digit ma¬ 
chine addresses. We’re not doing this any more. So broaden 
and deepen your understanding so you can distinguish be¬ 
tween good work and bad work in your sub-specialty of 
computing. 

DEFINITIONS 

Let’s define our terms. We are talking about training, not 
education. Education is the business of elementary schools, 
secondary schools, and undergraduate liberal arts programs. 
In my opinion, the goal of education is to broaden the whole 
person, to develop logical thought processes, mental and 
physical discipline, communications skills, self confidence, 
broad awareness, and appreciation. In short, education pre¬ 
pares one for training. Without education, training can be 
self-limiting. Pre-med is education, medical school is train¬ 
ing. Medical school prepares a doctor to diagnose illness or 
perform surgery; but pre-med enables him to grapple with 
concepts such as “health care delivery systems.” 

There is a growing trend in the lower schools that troubles 
me—the introduction of “computer labs” in secondary 
schools, and even in primary schools. Teachers are proud 
of turning their students on to computing at such tender 
years. No wonder the kids are excited and involved, and 
have to be peeled away from the terminals when the lab 
session is over. Programming and game playing are fun. 
Education is not. I was recently a guest lecturer to such a 
group of bright twelve-year-olds. They listened politely as 
I discussed how computers are used in banking—but when 
the compulsory lecture was done, they bowled me over to 
be first on the terminals. To be active, not reflective, and I 
am afraid, to do rather than think. Shouldn’t these teachers 
be more concerned about kids’ education than their training? 

In a similar view, Sesame Street was designed to hook 
disadvantaged kids and those who had no tradition of study¬ 
ing. It showed them that learning can be fun. It hooked lots 
more kids and blurred the distinction between learning and 
fun; teaching them, by inference, that if it isn’t fun, don’t 
bother learning it. Not so. Education can be drudgery. Te¬ 
dious hours of studying the classics. Is this why today’s 
Johnnie can’t read? 

What is professionalism? The answer, as Tevya said, 5 is 
very simple. I don’t know. But the main components are: 
defined body of knowledge of high intellectual content, de¬ 
fined standards of competence, examinations, code of eth¬ 
ics, and disciplinary capability. 6 These are present to a 
greater or lesser extent in the field of information systems. 
But to be a professional, you must first consider yourself a 
professional. 

Then what is professional development? It is the contin¬ 
uing technical training necessary to attain or maintain 
professionalism. At present, podiatrists in New York State 
are required by law to take 35 hours of training and be 
recertified bienially. There is quite a stir among the legal 
profession in the same regard. The state is now considering 7 
whether to require periodic refresher courses in four profes- 
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sional fields: those of physician, dentist, and optometrist, in 
addition to podiatrist. 

I do not believe that training in managerial skills falls 
within the boundaries of professional development. The for¬ 
mer is very important for those in management or aspiring 
to it but it is a different sort of problem and will not be 
addressed here. 


OPPORTUNITIES FOR PROFESSIONAL 

DEVELOPMENT 

So much for defining terms. Now to investigate opportuni¬ 
ties for professional development. Let us consider degree 
programs, employer-provided training, and self-develop¬ 
ment efforts. 

Degree programs 

Technical degree programs may be considered the highest 
form of professional development. On the other hand, grad¬ 
uate business programs are aimed more at personal or man¬ 
agerial development, and are not within the framework of 
this discussion as defined earlier. 

In passing, it is interesting to note the proliferation of new 
degree programs in Information Systems. Nunamaker 8 
found in a recent survey that 77 institutions are offering a 
bachelor’s or master’s degree in IS, MIS, MS, or some 
variant. Apparently universities are finding wisdom in the 
excellent report by Bob Ashenhurst. 9 The content of these 
programs is positioned between business and computer sci¬ 
ence, and in my opinion, within five years will surpass both 
the MBA and MCS in importance. One department head I 
lunched with recently is so besieged by inquiries from in¬ 
dustry for his graduates that he is considering retaining a 
telephone answering service. 

Degree programs in computer science provide a lot of 
theory and analysis, and a little skills training. They are at 
the leading edge of such disciplines such as systems software 
for minicomputers, but they lag the user community in oth¬ 
ers, for example manipulating large data files. Thus an eve¬ 
ning degree program in Computer Science may be under¬ 
taken as professional development by an applications 
programmer who aspires to work in systems software, rather 
than in systems analysis or management. 

University Computer Science programs, while theoretical, 
provide the student the basis for understanding (and ren¬ 
dering value judgments) on technological advances. So a 
few evening university courses from the Computer Science 
curriculum may be a useful means of professional develop¬ 
ment even for someone not wishing to pursue the degree. 
Many colleges and universities offer Computer Science ex¬ 
tension courses in non-campus locations for the convenience 
of computing professionals who are employed full time. As 
just one example, the New School for Social Research in 
New York offers a wide selection of evening certificate 
courses in computer science with a practical orientation— 
such as OS Debugging and Advanced Systems Analysis. 


Most employers pay or reimburse tuition for job-related 
courses if good grades are attained. Bell Laboratories and 
other enlightened companies, as well as universities, may 
grant sabbaticals to staff members to pursue advanced de¬ 
grees or research projects. There are a wide variety of com¬ 
puter-related professional development programs available 
thru colleges. The student must anticipate a major invest¬ 
ment of time to attend class and to study—time away from 
home and family—but as with a financial investment, the 
return is proportional to the risk. 

Employer-provided training 

The employer benefits in many ways from offering good 
technical training to its computing staff. Obviously, techni¬ 
cally proficient people work smarter instead of just harder. 
But also, turnover is reduced because professionals want to 
stay with an organization that offers technical growth. Re¬ 
cruiting is easier for the same reason. 

The mere availability of in-house training facilities is not 
enough. It remains for each individual to take advantage of 
the opportunities afforded him. Indeed, the company should 
adopt an active rather than passive role by scheduling and 
monitoring attendance. 

Technical training takes many forms depending upon size 
of company, rate of change in its computer installation, and 
degree of enlightenment. In a very large company or one 
with rapid growth or high turnover, classroom training is the 
mainstay. In a medium-sized or more stable installation, 
self-study courses on videotape are appropriate. And smaller 
or less advanced shops may rely strictly on sending people 
to external courses when and where they are offered. There 
is a place for all these modes of professional development 
in the typical company. If yours does not offer formal train¬ 
ing, lobby for its adoption. Here are basic guidelines for an 
internal Professional Development program. 

Establishing in-house training 

It is best that job descriptions and responsibilities be for¬ 
malized. If written job descriptions do not exist it could take 
many months until they are constructed to everyone’s sat¬ 
isfaction. In any event, knowledge requirements must be 
defined for each position whether from job descriptions or 
from hearsay. “Before you prepare instruction, before you 
choose material, machine or method, it is important to be 
able to state clearly what your goals are.” 10 Must Project 
Managers use Critical Path Scheduling Techniques? Are 
Senior Analysts responsible for discounted cash flow, or for 
statistical sampling methods? Do Lead Programmers pre¬ 
pare Warnier charts? 

Next, sources for each level of knowledge are identified 
by reviewing multimedia courses or evaluating vendor 
courses. Alternatively, specific courses can be developed 
in-house—but there are three disadvantages to such courses. 
First, they are very expensive. For an experienced teacher 
to develop course material can require ten hours of prepa- 
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ration time for each hour of classroom time. For an inex¬ 
perienced teacher working on new material the ratio can 
jump to 100 hours to one! Second is the effect of obsolesc¬ 
ence. Consider a course in direct-access file design. For the 
course to be useful for programmers and technical analysts, 
it should be quite detailed and relevant to the facilities avail¬ 
able. Student exercises should be directed toward the pres¬ 
ent installation. But when the direct-access devices or the 
installation access methods change, such training is instantly 
rendered out of date. The paradox is—the more specific the 
in-house course, the shorter its useful life. And a third dis¬ 
advantage is the difficulty of assembling enough students 
and a qualified instructor in one place at one time. 

Whatever the source of recommended courses, each job 
should have a list of recommended knowledge associated 
with it. Then present skills are rated and inventoried. Each 
member of the professional staff is asked to judge his pro¬ 
ficiency in the technical skills important to this job—for 
training purposes only. New hires are asked to rate them¬ 
selves similarly. Now, with skills required and skills avail¬ 
able known, the training required by each individual can be 
planned in a cooperative effort between staff member and 
his supervisor. Another valuable resource in such a meeting 
is the systems project plan, which identifies the assignments 
each person will work on. These meetings to plan training 
should be held at least once a year. It is during such a 
meeting that the staff member should communicate his own 
goals for specific technical training. And it is very important 
that training be scheduled for specific times and should be 
considered just as important as project work. Training must 
not be allowed to slip because of problems on another as¬ 
signment. Progress and performance in training must be 
measured and reported to management. It is not fill-in work 
and is not so treated by far-sighted organizations. 

As stated previously, about five percent of available time 
should be spent in training. And about two percent of the 
department’s salary budget should be allocated to out-of- 
pocket training costs. This sum will typically cover the cost 
of purchasing one or two multi-media courses as well as 
sending a number of people to external courses. Of course, 
the larger part of in-house training should use existing class¬ 
room or video courses at no out-of-pocket cost. The orga¬ 
nization that is just launching in-house training should con¬ 
sider a rental agreement rather than purchasing courses 
outright. A more elaborate discussion of the entire in-house 
training process is available 11 and will not be repeated here. 

Employer-provided external training 

External training takes many forms. Obviously each ven¬ 
dor offers courses in use of its hardware and software. 
Unfortunately, some of the mini-computer vendors are suf¬ 
fering from growing pains as their marketing success sur¬ 
passes their training capability. 

Many private training firms offer good courses in systems 
methodology. Brandon Systems Institute, Ware Associates 
of Hudson Massachusetts, and Control Data Corporation’s 
Institute for Advanced Technology come to mind. 


One may measure the state of the economy by counting 
the number of such brochures that fill the in-basket. In lean 
times, external training is the first budget item cut by most 
user companies, so the provider companies go into sus¬ 
pended animation. 

Even employers who make extensive use of in-house 
training facilities will continue to send selected people to 
external seminars. There is value in having Project Man¬ 
agers, for example, meet and socialize with their peers from 
other companies. They may pick up much of value from the 
successes and failures of other companies. And they may 
find out that things aren’t so bad in their own shop after all. 

The alert installation will collect, study, and file, written 
evaluations of all external (and internal) courses to be used 
in future planning. And the progressive installation will write 
the training firm to convey negative or positive views by 
attendees. This is appreciated by the vendor and helps to 
improve the breed. 

Another form of external training that must be mentioned 
to this audience is the Professional Development Seminars 
offered by ACM, IEEE, DPMA, ASM, and other computer- 
related professional societies. Such seminars are offered 
both at the national and local chapter level. They are usually 
aimed more at benefiting members than at making money, 
and are usually less expensive and may be less formal than 
for-profit courses conducted by nationwide firms. Earlier in 
this paper, an annual training plan was discussed. Profes¬ 
sional Development Seminars are rarely announced that far 
in advance. So the burden is on you to bring to the attention 
of your management any external PD seminar that supports 
your personal professional development plan. If no funds 
are available to pay the fee, ask for time off and pay it 
yourself. It will probably be tax-deductible. Under a liber¬ 
alization of regulations, costs of training required to maintain 
or improve skills on the present job are a deductible expense 
(but training for a new job is not). 

SELF-DEVELOPMENT EFFORTS 

We have touched on the professional development offer¬ 
ings of universities, and have seen what employers can and 
should provide. Now let us consider what the individual can 
do for himself. 

Establish a self-development plan 

Consider your circumstances to the following questions: 

1. Training taken in the last year 

2. Books read in the last year 

3. Current periodicals I read 

4. Tasks that I perform on my present job 

5. Personal qualities important on my job 

6. Changes that I can see in the future 

7. My strengths and weaknesses 

8. Goals within one year 

9. Goals beyond one year 

10. Specific plans to accomplish these goals 
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Such plans might resemble the following: 

(a) Read Weinberg’s An Introduction to General Systems 
Thinking before September. 

(b) Subscribe to Computerworld and EDP Analyzer; can¬ 
cel subscriptions to Sports Illustrated and Gourmet. 

(c) Begin a word list of all unfamiliar terms and their 
meanings. Carry this list and refer to it. 

(d) Discuss ideas found in technical publications with co¬ 
workers. Practice active listening by restating your 
associate’s view, in your own words, to his satisfac¬ 
tion. 

(e) Find a professional development seminar on distrib¬ 
uted data base. 

(f) Seek an opportunity to give a short guest lecture on 
minicomputer applications in process control. 

You should monitor progress on this plan and renew it 
periodically, at least annually. It is up to you; no one can 
do it for you. 

Sources of professional self-development 

Learning preferences and abilities differ. In our profession 
are found scholarly people and those who have come “thru 
the ranks.” We differ in age, in temperament, and in expe¬ 
rience. Thus training methods should differ. Some students 
respond favorably to video courses while others find them 
boring and would prefer to learn by studying reference man¬ 
uals. Sources of the latter are well-known, and fortunately 
the former have also been documented. 13 The IEEE offers 
a wide variety of self-development opportunities to mem¬ 
bers, although not all in computers. For example there are 
self-study courses in engineering, on such subjects as Field 
Effect Transistors, and Digital Instrumentation; self-study 
courses in management, for example How to Start Your 
Own Consulting Business; approximately 100 field courses 
(to be held by your company); correspondence courses; and 
speed reading. 

Many people learn by doing. “What I hear, I forget; what 
I see, I remember; what I do, I know.” How to better learn 
about microcomputers than to build one? An IMSAI 8080 
IK kit currently sells for $650, a saving of $350 over the 
assembled price. An expensive and time-consuming lesson, 
but one guaranteed to produce expertise and build a foun¬ 
dation for the future. (The 8080 chip, by the way, came out 
in 1974 priced at $300 in OEM quantities—at this writing. 
Radio Shack sells it over the counter for $18!) 

Microcomputers may very well provide the at-home 
professional development wave of the future. Computer-as¬ 
sisted instruction has not yet become cost-effective except 
in special situations. But the Video Disk technology has 
prospects of providing huge quantities of read-only software, 
including course material, to microcomputer users—elimi¬ 
nating the traditional CAI requirement for large CPU con¬ 
nect time. The ACM Professional Development Committee 
is watching developments carefully. 


Establish a reading program 

Beyond any questions, the computing literature is over¬ 
whelming. The ACM Computing Reviews subscribes to 240 
periodicals! Only a small fraction of their articles, and only 
a limited number of textbooks can be reviewed by Comput¬ 
ing Reviews. But their staff of editors and reviewers does 
an admirable job—and the publication can be of great benefit 
to the reader. Clip or copy reviews of importance to you. 
File them and track them down when you have time or when 
your need is more urgent. Become a source of resource 
material to your associates. How can anyone be considered 
a professional who does not stay conversant with the liter¬ 
ature of the field? 

Each of the professional computing societies has a lead 
journal, most also have many special interest journals. They 
are far too numerous to list. Everyone ought to pick a few 
favorites and set aside time to scan them, marking or clip¬ 
ping articles to be read. Mine are Datamation, Computer- 
world, EDP Analyzer, ACM Computing Surveys, IEEE 
Computer, the IBM Systems Journal, and Scientific Amer¬ 
ican. The September, 1977, issue of Scientific American, 
devoted to microelectronics, was a revelation to me. If suf¬ 
ficiently enlarged and studied, the Boolean logic of SLT 
circuits can be seen! 

The Honeywell Journal, edited by Bob Berner, had a fine 
format. It included microfiche in the inside cover, was multi¬ 
lingual in part, was printed on fine paper of the European 
standard size, and published significant articles with high 
quality graphics. Unfortunately it perished in the economic 
crunch a few years ago. 

Take a rapid reading course and practice what you learn. 
One MBA student, overwhelmed with reading assignments, 
sat under the clock in the library and turned a page every 
time the clock’s minute hand clunked. Even if he had not 
quite finished both sheets, he had the sense of the material. 
And 60 pages an hour is not a bad pace. Technical material 
may require more time, particularly if you own your books 
and write comments in the margin. There are several book 
clubs for computing professionals. 12 Membership will force 
you to set a reading schedule. However, many people be¬ 
lieve in selecting their own books based on published re¬ 
views and personal recommendations, rather than reading 
books selected by committee. Discuss your reading and your 
opinion of it with your friends. 

Other individual approaches 

Your attendance here suggests two things: that you are a 
member of one of the AFIPS constituent societies, and that 
you are interested in professional development. If so, attend 
your evening chapter meetings. Practice your active listen¬ 
ing skills on the speaker. Take notes (“what I do, I remem¬ 
ber”) and ask questions. Use the coffee or cocktail session 
to learn more, not to tell how you blasted out of a sand trap. 

It is difficult, but useful, to find out what your coffee 
partner knows that may be interesting to you. Recently I 
learned, from a stranger, that the IRS master file consists of 
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1300 reels of tape! Such hearsay may not be totally reliable, 
but as a computing professional you have learned to render 
value judgments on incoming data. 

Keep an audio cassette player in your car. Listen to 
speeches 14 you were unable to attend. Note-taking is a little 
more difficult here. 

CONCLUSION 

This paper has presented a variety of approaches to profes¬ 
sional development. They range from formal university pro¬ 
grams, to the semi-formal company courses, and the infor¬ 
mal, personal endeavors in technical training. It should be 
apparent that there is enough variety and quantity of options 
that no one need to suffer from the malaise of creeping 
technical obsolescence. But it is an individual decision to 
construct an action plan and to persevere. No one can do 
it for you. Good luck! 

DISCLAIMER 

The citations given in the text and reference list were se¬ 
lected by the author. No attempt has been made to exhaus¬ 
tively list all training vendors, publishers, etc. The reader 
may use his resources to locate others, or is invited to 
contact the author for further information. 
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Professionalism in data processing management 


by DELBERT W. ATWOOD, JR. 

Utah State Board of Education 
Salt Lake City, Utah 


INTRODUCTION 

Where do you stand as a member of the information proc¬ 
essing community when the discussion turns to defining data 
processing as a “profession?” (Webster says, “One that 
engages in a pursuit or activity professionally.” Webster 
also says, “Characterized by or conforming to the technical 
or ethical standards of a profession.”) Professionalism is, 
“The conduct, aims, or qualities that characterize or mark 
a profession or a professional person.” Black’s Law Dic¬ 
tionary defines profession as: 

A vocation, calling, occupation or employment involving 
labor, skill, education, special knowledge and compen¬ 
sation or profit, but the labor and skill involved is pre¬ 
dominantly mental or intellectual, rather than physical or 
manual. The method or means pursued by persons of 
technical or scientific training. 

Black includes the following clarifying paragraph for the 
definition of a profession. 

The term originally contemplated only theology, law and 
medicine, but as application of science and learning are 
extended to other departments of affairs, other vocations 
also receive the name, which implies professed attain¬ 
ments in a special knowledge as distinguished from mere 
skill. 

Occupational sociology literature suggests that modern 
technologically oriented societies indicate that the distinc¬ 
tion once made between “Professional” and “Non-Profes¬ 
sional” is blurred or vague. 

ATTRIBUTES OF A PROFESSIONAL 

In my opinion, the attributes of a professional are many. 
The qualities exhibited by a data processing professional 
place him/her above the “run of the mill” person. The 
professional: 

• is knowledgeable, yet not overbearing 

• listens, but does not always comment 


• offers advice, but does not dictate 

• works with people 

• constantly strives to maintain technical competency 

• is respected by his/her actions 

• has extensive training in a specialized discipline sup¬ 
ported by a body or research and theory 

• supports a “code of ethics” developed by an associa¬ 
tion of colleagues 

• performs a unique and essential service recognized by 
the community 

• exhibits strong personal responsibility for his/her judg¬ 
ments and actions. 

Individuals who score high on a professionalism mea¬ 
sure exhibit a strong desire for implementing professional 
values among their peers. 

One of the largest obstacles of any attempt in defining 
professionalism in data processing management is the lack 
of definite criteria as to what makes up a “professional” 
computer person. Some of these criteria might be: 

• Education 

• Knowledge 

• Ethics 

• Communication Abilities 

• Public Relations Abilities 

As we look back on Webster’s definitions and characterize 
them as to our own particular situation, can we say that we 
are truly a professional, or merely standing on the outside 
looking in. Do we belong just for the sake of belonging, or 
do we belong so that we can be recognized as an individual 
who is knowledgeable about the profession and can be help¬ 
ful to our management and companies as a result. 

Leonard I. Krauss, in his book, Administering and Con¬ 
trolling the Company Data Processing Function, has a sec¬ 
tion on professionalism in the field. He states, “There are 
a number of factors which are characteristic of a profession. 
Rather than undertake a debate on whether data processing 
is a profession, the time could be better spent by discussing 
professionalism as it pertains to data processing. The aims 
and standards of data processing have been the concern of 
several data processing organizations, and also the USA 
Standards Institute. ” 3 
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HOW TO ACHIEVE PROFESSIONALISM 

A manager achieves a professional status from the actions 
of his peers and those who work for him. He is respected 
for his decisions. He is highly regarded by those who work 
for him and also those for whom he works. When a man is 
looked to for advice and counsel, and has gained the respect 
of others regarding the many aspects of the business he is 
engaged in, he is on the way to achieving the professional 
qualities described above. An individual can enhance his 
professional appearance through his training, education, as¬ 
sociation with others, his actions on the job as well as his 
personal life, and the activities to help others. 

We, as managers of the information services areas of our 
companies or shops, have an obligation to our company, 
our management, our fellow employees, our staff, and also 
to ourselves to help make people understand our methods 
and procedures of operation, to advise our management so 
that sound, honest decisions can be made regarding our 
realm of responsibility. We must educate ourselves to be 
knowledgeable in the area of our pursuit or activity. We 
must work together to disseminate our knowledge to fellow 
workers and management, to realize the maximum benefit 
from our education. We must be loyal to our companies or 
firms, respecting the trust vested in us by them. We must 
conduct ourselves in an ethical manner at all times regarding 
the information processing profession and to any profes¬ 
sional organization to which we belong. Your attitudes and 
actions mold for you the things that you are and the way 
other people see you. 1 

Are you a professional? Do you belong to a professional 
organization? George R. Terry, in his book Principles of 
Management , states “Active membership in a truly profes¬ 
sional group can assist management development through 
attendance at meetings, informal discussions with other 
members, and the reading of the association’s journal or 
official publication. Some associations stress individual self¬ 
development; others promote workshops, seminars and 
study groups. These experiences help to promote the im¬ 
portant “develop yourself philosophy.” Generally, most 
companies or employers wish to have their employees the 
most knowledgeable individuals in regard to their expertise. 
In order to accomplish this, they send them to schools and 
allow them to belong to groups that enable them to meet, 
share ideas, experiences and needs. They, in most cases, 
pay for the memberships and expenses. Companies and em¬ 
ployers provide these additional fringe benefits to encourage 
professional development. This professional development 
should enable them to do a job better, quicker, and generally 
at a lesser cost. 

When you are knowledgeable about a subject, your spe¬ 
cialty, and you discuss and share this knowledge with your 
management, your peers and others, you become a profes¬ 
sional. This recognition comes from others and can be 
helped by your membership in a professional group and your 
participation in all of the activities the professional group 
can provide. 

Professionalism is sometimes associated with an honor 
society, or a certification of one type or another. This could 


also be in the form of an examination to license, to allow 
the individual to practice their specialty. This is character¬ 
ized by the state boards to allow an M.D. to practice; the 
Bar to allow an attorney to hang out his shingle; the C.P.A. 
examination to permit the accountant who passes it to certify 
that his examination of a set of books is appropriate accord¬ 
ing to standards and acceptable. 

Data processing, through DPMA, has constructed the cer¬ 
tificate in Data Processing (CDP) as a test of knowledge in 
management, systems quantitative and qualitative analysis, 
programming, and other areas. Many professional groups 
joined together to form the Institute for Certification of 
Computer Professionals (ICCP). These groups recognized 
the need for a professional certificate to assist the individual 
in his personal development, recognition by his peers, his 
company or employer, and his development as a profes¬ 
sional. 

Many of the organizations have by-laws, codes of ethics, 
guidelines for professional conduct in information process¬ 
ing, and other written vehicles for the individual to follow 
to assist him in becoming a professional. Adherence to these 
precepts certainly would help us to be recognized as follow¬ 
ing a professional path. 

Where do you stand in your own professional develop¬ 
ment and standing in your company, and within the orga¬ 
nization to which you belong? 

Are you going to classes to get your degree? Are you 
getting an advanced degree? Are you attending seminars to 
further your knowledge in your specialty subject matter? In 
our ever expanding field where specialized knowledge is 
mandatory, those individuals who demonstrate superior 
competence in the application of that specialized knowledge 
reap the rewards of progression. 

The individuals who are looking for management recog¬ 
nition and who are looking for a greater professional posture 
in their work, may be selfishly motivated, but in the long 
run, they are on the road to becoming a professional. The 
knowledge and association developed in the process gener¬ 
ally stimulates the individual to be more reflective about his 
work, and approach problems with a greater understanding 
and to bring to bear the new techniques and concepts learned 
to satisfy those problems. 

Do you do your part? Are you becoming the professional 
your management would like you to become? Do you belong 
to a professional group, but never go to the meetings? You 
read they have excellent seminars and workshops, but when 
was the last time you attended one of them? Do you get a 
publication from the association? When was the last time 
you read it from cover to cover, or even selected articles? 
Can you quote from any of the articles about a particular 
subject matter that is coming up in a management committee 
meeting tomorrow? Can you refer to other publications, 
books, seminars, workshops, etc. to assist in helping to 
make management decisions? 

Professionalism in data processing is beginning to emerge. 
It still may take a while to accomplish, but the tools are 
there, and if the right person or persons come along to guide 
the tools, the guidelines, ethics standards, methods of con¬ 
trol and other details will be pulled together and be recog- 
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nized by all information processing personnel and organi¬ 
zations, and will be thus enforced by them or their peers 
who make up the organizations. The profession is growing 
by leaps and bounds, and some restraint or control is needed 
before it gets out of hand. 

CONCLUSION 

I believe that professionalism makes a difference, because 
an individual who is looked up to, respected by his associ¬ 
ates and others, who exhibits the traits that place him a bit 
above, has a better chance for success and overall happiness 
in his life. A doctor, a lawyer, or an engineer work very 
hard to become educated and licensed to practice their spe¬ 
cialty; likewise, data processing managers work very hard 


to become educated and knowledgeable in their specialty. 
Licensing is a way off, but maybe it will become a necessity 
before professionalism is a reality, certification is here now 
and is a step toward achieving the recognition and respect 
of our employers, our employees, and our peers. 
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Professionalism—A question of semantics 


by EUGENE B. SMITH 

U.S. Department of Agriculture 
Belts ville, Maryland 


INTRODUCTION 

The topic of professionalism is of crucial importance to the 
ever-increasing group of individuals dedicated to the design, 
production, dissemination of knowledge, and application of 
the computer. There is a strong desire for professional rec¬ 
ognition by large portions of this group. This paper focuses 
on the need for an appropriate definition for a computer 
professional. Questions are raised concerning the various 
requirements and related responsibilities which should be 
associated with recognition as a professional. The element 
of controversy over numerous issues will serve to clarify, 
if not resolve, the parameters which outline the future data 
processing professional. 


PROFESSIONALISM DEFINED 

In order to establish a baseline from which this disserta¬ 
tion can proceed, there is some value in presenting a “Web¬ 
ster’s” definition of terms which are of fundamental impor¬ 
tance to the topic. These terms include the following: 

• Profess—To practice or claim to be versed in (a calling 
or profession). 

• Profession—A calling requiring specialized knowledge 
and often long and intensive academic preparation. 

• Professional—Characterized by or conforming to the 
technical or ethical standards of a profession. (Partici¬ 
pating for gain or livelihood in an activity or field of 
endeavor often engaged in by amateurs). 

• Professionalism—The conduct, aims, or qualities that 
characterize or mark a profession or a professional per¬ 
son. 

The term professionalism has different meanings to dif¬ 
ferent people. Ralston 1 defined professionalism by saying 
that ”... a professional is someone dedicated to rigid train¬ 
ing rather than immediate self-gratification. As such, 
“professional” bespeaks not elitism but a state of mind.” 

In an attempt to define the term professional by example, 
it would seem that there are at least two distinguishable 


groups. The first group might be called the “learned” 
professional and it might be categorized as having demon¬ 
strated a profound knowledge or scholarship. This group 
includes the universally recognized titles of doctor, lawyer, 
and engineer. Their professional recognition is based on an 
attained level of knowledge in a specialized field; a certifi¬ 
cation that they have reached a specified level of knowledge; 
and legal registration which gives them the right to practice 
their profession within certain constraints. 

A second group of professionals may be categorized as 
the “other” professionals. These individuals may be iden¬ 
tified as being associated with the second definition for 
professional which was given previously. That is, we may 
consider that they participate for gain or livelihood in an 
activity or field of endeavor often engaged in by amateurs. 
This group would include the professional bowler, the 
professional boxer, the professional football player, the 
professional golfer, and others. In addition to professional 
athletes, we might also include the professional politician, 
the journeyman electrician, and the journeyman plumber. 
The professional athlete has recognition based on his level 
of ability in his chosen field. In most cases this status is 
governed by membership in a professional association. The 
professional trades normally require an apprenticeship as¬ 
sociated with a trade union. Advancement to the journey¬ 
man level implies the attainment of a reasonable skill. In 
some cases, local governments require that a licensed jour¬ 
neyman be responsible for the quality of work performed. 

It may be helpful to associate the “learned” and “other” 
professionals on a continuum. In terms of respect or service 
to society, we may place the “doctor” at the high end of 
the continuum. In terms of income or monetary reward, we 
would have to place the boxer at the first of a continuum. 
In the case of either the “other” professional or the 
“learned” professional it is possible to identify the creden¬ 
tials associated with their professional status. In the case of 
the computer professional, it is important to identify the 
credentials associated with the achievement of a profes¬ 
sional status. In addition to other things, these credentials 
are meant to insure a minimum level of competency and 
adherence to certain standard practices. If suitable creden¬ 
tials or criteria do not exist, it is important that these be 
identified or established to the satisfaction of the individual, 
his peer group, his employer, and society at large. 
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ELEMENTS OF DATA PROCESSING 

PROFESSIONALISM 

In addition to identifying the various factors associated 
with the achievement of a high level of professional recog¬ 
nition in the field of data processing, it is important to pursue 
the parameters or limits associated with each factor. It must 
be possible for individuals to work toward achieving the 
status of a professional and know when they have obtained 
it. Such recognition must extend beyond the peer group to 
the employer and to society as a whole. Once a high level 
of professional recognition is possible, it may be that many 
individuals in data processing may not be suitably motivated 
to try to achieve it. 

Perhaps the first question to be resolved is why we wish 
to achieve a professional status. What is our purpose? Is it 
to satisfy some personal ego requirement which would allow 
one to be a member of an elite group? Do we wish to 
improve our monetary status within our group? Do we wish 
to improve the moral and ethical level of our peer group? 
Are we interested in improving our level of service to both 
our employer and society as a whole? We are associated 
with the use of a technology which can have awesome im¬ 
plications to all of mankind. It is important to give consid¬ 
eration to the underlying motivation for our desire for 
professional status. 

The term profession implies the existence of a certain 
level of specialized knowledge. One might question the 
make-up of this body of knowledge and how it can be vali¬ 
dated. The Institute for Certification of Computer Profes¬ 
sionals 2 identifies the content of the Certificate in Data Proc¬ 
essing (CDP) examination as containing sections on Data 
Processing Equipment, Computer Programming and Soft¬ 
ware, Principles of Management, Quantitative Methods, and 
Systems Analysis and Design. The CDP eligibility require¬ 
ments consist of the equivalent of 60 months full time related 
experience. An academic alternative allows a substitution of 
designated academic work for up to two years experience. 
This sets an example for a test that, to some extent, validates 
a level of knowledge at one point in time. Is this “the” 
common body of knowledge which the computer profes¬ 
sional should have? Are there several specialties as there 
are in medicine or engineering? 

If we could assume that this is “the” common body of 
knowledge that is desired and that the eligibility require¬ 
ments are suitable, there are still many other factors which 
must be considered. Is there a need for periodic re-certifi¬ 
cation? Both DPMA and ACM have Self-Assessment pro¬ 
grams under way which should help an individual identify 
his weak points and provide references for correcting them. 
Continuing education in many forms is available on almost 
any topic. The mechanism for maintaining a current knowl¬ 
edge of the state-of-the-art is present but the verification 
vehicle is missing. 

Consider the ethical implications of professional status 
and how they might apply to this situation. In terms of 
loyalty, Pletta and Gray 3 point out that “. . . the profes¬ 
sional’s primary loyalty may be to: (1) society; (2) his peer 


group; (3) employer or client; or (4) himself. Only if he is 
free to elect the first of these does society confer upon his 
peer group the status of a profession.” Loyalty will greatly 
influence a course of action when a question of conflict of 
interest arises. Questions involving privacy and security are 
also related to ethics. 

While each major professional group has a code of ethics 
associated with membership, one might question the ade¬ 
quacy of the requirements. It would seem that the enforce¬ 
ment provisions and mechanisms leave something to be de¬ 
sired. Even the old line professional groups such as the 
doctors and lawyers have great difficulty in adequately en¬ 
forcing their respective codes. 

The legal implications of professional status are primarily 
related to a licensing process. Municipal, state, and federal 
agencies often have a role in licensing. The various legisla¬ 
tive groups must enact specific laws to create the license 
mechanism. Lord 4 goes to great lengths to distinguish be¬ 
tween the licensing and certification processes. The licens¬ 
ing process must also have a sound basis for enforcement. 

It is appropriate to identify the various players and their 
role in the question of professionalism. The list includes the 
following: 

Academia —The majority of the formal training will be 
done in the numerous colleges and universities. These in¬ 
stitutions have a responsibility to provide training in an 
accepted common body of data processing knowledge. The 
curricula must be kept current with the changing technology. 
In addition to a well rounded education they must attempt 
to instill a sense of responsibility, morality, ethics, and gen¬ 
eral professional awareness in the students. 

Professional Groups —The role of continuing education 
carried out through professional meetings, quality publica¬ 
tions and innovations such as the self-assessment programs 
is a primary responsibility of these organizations. Their role 
in certification, and/or some accepted vehicle for determin¬ 
ing the minimum level of knowledge is crucial. An organi¬ 
zation such as the ICCP, which receives wide support from 
most related professional groups, is needed to serve as the 
testing and enforcement agency. The professional group 
must strive to represent and provide a unified voice for the 
membership. 

Employers —If a significant professional status is to be 
achieved, it must be recognized and supported by business 
and industry. This recognition could take the form of con¬ 
sideration in the hiring and promotion processes. 

Governmental Agencies —In the event that some form of 
licensing or registration is appropriate, the government 
agencies would be expected to establish the requirements, 
procedures, and control mechanisms for enactment of leg¬ 
islation. 

Society —If society does recognize a professional group 
through general acceptance and the enactment of legislation, 
it has the right to expect a high level of moral and ethical 
responsibility. Society also has the responsibility to speak 
out when the actions of the professional adversely affect 
segments of the public. 
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The Individual —When an individual achieves the status 
of a professional, he must accept certain responsibilities 
along with the related benefits. Some of these responsibili¬ 
ties are legal and some are social. In the event that he fails 
to meet the responsibilities, his peers must be willing to take 
appropriate corrective action. 

Each player has a role and associated responsibilities in 
progress toward professional recognition for the Data Pro¬ 
cessor. The failure of any player to assume his proper role 
and responsibility will impede the progress. 

SUMMARY 

Professional status will be achieved in the data processing 
area. The speed and degree to which this is accomplished 


will depend on the dedication of the individual and his level 

of support for the data processing professional societies and 

related groups. 
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Continuing education opportunities—A mark of a profession 

by ROLAND D. SPANIOL 

Eastern Illinois University 
Charleston, Illinois 


INTRODUCTION 

Authorities indicate that one of the marks of a profession is 
the availability of continuing education to its members. Ed¬ 
ucation that will tend to keep the members current with their 
technology and skilled in applying it in their work. It seems 
reasonable this should be a requirement of information sys¬ 
tems if it is to be designated a profession. 


AN INTERPRETATION OF PRESENT CONDITIONS 

Educational products for people in information systems 
occupations come from a variety of sources including hard¬ 
ware manufacturers, software houses, government regulated 
industries, universities and colleges, profit motivated 
schools, businesses, etc. Based on the large number of ad¬ 
vertisements in the mail each day, it is easy to identify the 
variety of educational offerings that are available. There are 
many; they are offered in many big cities; and they cover a 
wide range of topics. 

The present situation with respect to educational offerings 
is summarized in the following paragraphs. 

The enormous number of group and self-instructional 
courses presently on the market cover the majority of the 
subject areas revealed in the DPMA Education Foundation’s 
“Survey of Educational Needs and Desires.’’ 1 For example, 
25 of the 34 most frequently mentioned subject areas in the 
survey are covered by one or more current promotional 
brochures. 

Is the demand for these courses so great that the needs of 
the information systems practitioners cannot be met? Is the 
demand being driven by a desire on the part of the infor¬ 
mation systems practitioners to be professional? Do you 
think the courses would be as heavily advertised as they are 
if this were the case? In short, most of what the information 
systems practitioner says he needs is probably available; but 
to be blunt, much of it is probably the “paper tiger” variety. 

At the present time prospective registrants have no fea¬ 
sible way of evaluating the quality of an offering beyond the 
general reputation of the firm that markets it or the personal 
recommendation of past participants with whom they may 
have had some chance contact. 


Many private educational offerings are priced well beyond 
the reach of those who probably need them the most. This 
unfortunate situation is due, of course, to the high cost of 
producing and delivering an educational product in combi¬ 
nation with too few buyers spread over too many sellers. 

To summarize present conditions, it can be said that most 
of what information processing professionals say they need 
is available. It may not be in the precise form they would 
like it to be, but it is available. Their problem is to find it 
among the plethora of offerings while avoiding poor quality 
and figuring out a way to pay for it. 

If it is true that the information systems practitioner’s 
educational needs can generally be met with current offer¬ 
ings in a variety of big cities, but that the cost is high, what 
is the solution? The one obvious solution that comes to mind 
is to cut the number of offerings, teach to a larger group at 
a time, and lower the tuition for each student. Those of you 
involved in higher education in recent years will note this is 
not an original solution. Those of you involved in offering 
the courses from the private sector may be curious how this 
can be accomplished. 


EVALUATION TECHNIQUES 

What I am suggesting is not new either, it is supply and 
demand aided with market analysis, evaluation of the offer¬ 
ing, publicity, and a booking service. 

Market response to an educational offering is a good in¬ 
dicator of the worth of the offering. If the course is popular 
and makes money, it can be repeated; when it loses its 
appeal, enrollments decline, and profit wanes, it will be 
dropped. This is not an unknown phenomenon (except 
maybe in public education). As a matter of fact, for educa¬ 
tional offerings from the private sector it is principally the 
law of supply and demand that decides whether a course is 
to be repeated. Demand and enrollment are the required 
measures of the quality of the offering. 

Reaction to an educational offering is a commonly used 
indicator of the quality of the offering. The technique is 
familiar to you; the attendees each receive an evaluation 
form near the end of a course and the attendees rate the 
presentor, the presentation, the content, the facilities, etc. 
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Since it is easy to do and results are quickly available, it is 
often used with market response as the way to improve the 
offering, to determine whether or not it should be offered, 
which instructor is getting the best evaluation, type of fa¬ 
cilities most desired, etc. Later, you will see that it leaves 
room for improvement as an evaluation instrument, but it is 
an important measure of how well the program was ac¬ 
cepted. 

Learning that has taken place because of an educational 
experience is a better measure of an educational offering. 
For this situation, suffice it to say that learning is the ability 
to recall a concept, knowledge, technique, or item of infor¬ 
mation in a testing situation. Comparing reaction to learning, 
it can be seen that learning is a better measure of the success 
of a program than reaction because with learning we know 
that the program message is getting through to the enrollees. 

Learning is more difficult to measure because if we are to 
be precise, we must test before we teach and then test after 
we teach so we can develop a statistical correlation and/or 
confidence level. Because it is more difficult may be the 
reason we rarely see testing done in courses offered by the 
private sector. What do you think? 

Behavior changes in an individual that are perceptible to 
others as the result of an educational experience is an ex¬ 
cellent measure of the value of that experience. But, meas¬ 
uring behavior changes is much more complex, difficult, and 
costly than measuring learning or reaction. It usually re¬ 
quires experts in testing, is usually administered to a special 
group or groups at a time, is usually done on contract, etc. 
Though difficult, measuring behavior changes is certainly a 
valid evaluation process and should be considered in special 
situations. 

Results is what everyone wants. Why not measure the 
contribution of educational programs based on whether the 
enrollees get satisfactory results from implementing their 
new found knowledges, techniques, skills, etc.? Probably 
because results are more difficult to measure than market 
response, reaction, behavior, and learning combined. Not 
only that, but it is difficult to be sure that the results 
achieved are related to the instruction. Much more work 
needs to be done relative to measuring results. 


IMPLEMENTATION OF A NATIONWIDE PROGRAM 
Catalog of programs 

Initial efforts should be directed at facing the formidable 
tasks of identifying programs and materials currently in exis¬ 
tence that will meet the members professed educational 
needs. Indeed, the greatest contribution may be in seeing 
that an ongoing cataloging process is set in motion to identify 
to the membership where and when they can enroll in pro¬ 
grams they feel they require to keep them abreast of current 
technology. A catalog of these program offerings could be 
made available on a subscription basis to libraries, busi¬ 
nesses, individuals, etc. In order to be kept current it would 
have to be updated no less than monthly. The listings should 


be complete with course outline, presenter, dates, places, 
costs, etc. 


Booking service 

A natural byproduct of a catalog of programs is a nation¬ 
wide booking service, a centralized office where persons 
could enroll in the courses of their choice. This office could 
be the central point for all tuition, enrollments, evaluation 
activities, etc. 


EVALUATION PLAN 

Evaluation of the programs cooperating in the project 
would be a major activity. It is recommended that reaction 
evaluation be the first method used. This method should be 
used for all educational offerings made a part of this service. 
The courses in the top 10 to 25 percent should be given 
special publicity such that they would be widely recognized 
as popular courses. 

As the project matures, the testing for learning process 
should be initiated. Attempts should be made to see if learn¬ 
ing does take place. In the programs where learning is great¬ 
est, the courses should be given special publicity so they 
would become widely known as good programs to take to 
learn the needed information. 

In some special cases, attempts should be made to deter¬ 
mine changes in behavior. Because this evaluation method 
would be difficult to implement and would have marginal 
influence on this project, it will not be dealt with in this 
paper. 

Results evaluation is an interesting method and in a broad 
sense could make some real contributions on a nationwide 
basis. Initially, or in the foreseeable future, there does not 
seem to be much results evaluation can do for this project 
and will not be dealt with in this paper. 

On a nationwide scale, however, it would be interesting 
to see if the information systems professionals could identify 
the knowledges of the citizens about an issue such as pri¬ 
vacy. Then, after implementing a large scale public educa¬ 
tion program on the issues of privacy, follow this up with a 
post test. Enough said. 


WHO COULD MANAGE THIS CONTINUING 

EDUCATION PROJECT? 

Possibly the DPMA Education Foundation. Or maybe any 
of the professional organizations. How about AFIPS? Per¬ 
haps because of my prejudices, I think the DPMA-EF is 
best suited for the job. They are an IRS 501 (c) (3) not-for- 
profit corporation. They have no members, and they were 
organized with the major objective of providing education 
to the information systems practitioners. Why not let them 
know how you feel about having them sponsor such a proj¬ 
ect. 
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WHAT WOULD MOTIVATE COOPERATION WITH 

THIS PROJECT? 

Have you ever sat through a two or three-day course or 
a semester one for that matter and felt you were wasting 
your time? Do you think that experience was unusual? 
Would you like to do something about it? Of course, we 
have all had bad experiences on occasion. And, if there was 
a formal process where we could input our thoughts about 
a course, we would have done so. The primary motivation 
for an individual to cooperate in the evaluation process is 
improved education. The more information systems practi¬ 
tioners we can get to evaluate the course offerings they take, 
the better chance we have to weed out the less effective 
courses. 

Getting cooperation from the presentors may initially be 
more difficult than getting cooperation from the individuals 
taking the instruction. After all, what is in it for the presen- 
tor? Of course, it depends upon the management of the 
project, but I would hope the presentors would soon see the 
merits and become the proponents of the process. 

The project incentives which will engender presentor par¬ 
ticipation will be positive incentives. If the project is to be 
successful, the presentors must want to participate, must 
strive for the opportunity to be listed in the catalog, must 
encourage enrollees to participate in the evaluation, etc. 
What will encourage this eagerness? 

The possibles include the following: 

An honor roll of blue ribbon presentations. A panel of 
judges selected by the participating presentor organizations 
would select a limited number of offerings for special rec¬ 
ognition. The top two or three presentors would be selected 
from this group for a special award or prize. 

The opportunity of participating in a cooperative nation¬ 
wide publicity activity. Through advertising in the trade 
press and other publicity, the offerings of the cooperating 
presentors would be made available to the information sys¬ 
tems practitioners. 

A catalog of the offerings of all the presentors. The listings 
would be by presentor organization, by subject, by city, and 
by date, for example. This catalog would be made available 
on a subscription basis. The subscription price would be low 
enough to be attractive to public libraries, companies, and 
certain other training organizations, and governmental agen¬ 
cies. 


The opportunity of participating with experts in the proc¬ 
ess of developing evaluation techniques. It is anticipated 
that the number of presentors involved would be great 
enough that they could employ testing specialists to work 
with them on the evaluation processes. 

Altruistic as it may sound, I hope some of the presentors 
would want to participate for the simple reason that they 
would like to improve their offering. Just because they want 
to do a better job. 

Then, there are other kinds of reasons. Some information 
systems practitioners would like to show that they have the 
trappings of a professional. These might include a college 
degree, a certificate such as the CDP, and some would even 
prefer a license to practice. 

In any event, it seems reasonable that if a profession 
offers continuing education opportunities to its members, its 
members may want to have a formal method of showing 
they are taking advantage of this education. 

There is such a system where courses are evaluated and 
are then assigned whole or fractional units of instruction. 
These units are called Continuing Education Units and are 
known as CEU’s. It seems reasonable the programs coop¬ 
erating with this project could be dealt with in a similar 
fashion and the record keeping for the participants could be 
kept as part of the project. 


SUMMARY 

In this paper the surface of an idea has been presented. 
What we need now is someone who wants to develop this 
idea into a reality. I hope you will agree that there are 
several prevailing reasons for our industry to formalize and 
strive to improve the continuing education aspects of the 
industry. We in the industry will be the last to know we are 
in a profession or when we became a profession. Let’s make 
sure we have taken every opportunity to be as professional 
as we can be now that we are a part of that group. 
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INTRODUCTION 

The human ego provides continuous opportunities to explore 
the richness and variety of human perception. It is capable 
of excruciating truth and insight as well as delusions of 
grandeur that obscure truth and reality. The maturity of an 
individual can be defined in relation to his ability to differ¬ 
entiate between these extremes. 

The computing practitioner, being human, is no exception. 
His existence in a rampant technology substantially lacking 
established codes of standards, conduct, and practice ac¬ 
centuates his capability to delude himself and makes more 
difficult a clear understanding of the reality of his experience 
in a complex environment. He can to a large degree establish 
his own criteria of accomplishment, yet must feel some 
concern of exposure as he realizes the effects of the rapid 
change forced upon him by his technology. 

This paper postulates two resulting classes of behavior 
observed in the industry, one which could provide consid¬ 
erable benefit to the individual in coping with his anxieties 
while the other appears to be counter productive. 

The first category can be described as behavior reflecting 
a desire to be identified as a “professional” and as such be 
associated with practitioners characterized as possessing 
competence, integrity, contribution, and esteem. 

The second category represents a potpourri of behaviors 
generally designed to avoid exposure of the anxieties ex¬ 
perienced from the self-doubt engendered among those re¬ 
sponsible for the implementation and control of rampant 
technological change. 

The purpose of this paper is to investigate the current 
level of maturity in the industry which has been defined as 
differentiating between reality and protective delusions, and 
to make recommendations upon the conclusions drawn. 

One cannot but be impressed with the frequent and per¬ 
vasive use of the objective “professional” in reference to 
the many authors and speakers represented in the trade 
press and at various computing conferences. It appears that 
the proverbial “Everyman” considers himself a professional 
although considerable debate exists as to exactly what is the 
meaning of the term. In fact, one might express curiosity at 
the apparent egotism evidenced in the employment of the 
term. While clearly proposing their own professional status, 


most express serious doubts about the credentials of any 
others. What then are the essential criteria of a professional? 

To begin, professionalism in the computing industry can¬ 
not be understood unless one first clearly understands what 
a profession is generally and the environment in which the 
professional finds himself. 

The age in which we live has been characterized as one 
suffering from “Future Shock,” a term defined by author 
Alvin Toffler as the effects of “. . . too much change too 
fast.” As our world, environment, institutions, our very 
lives and personal values reel under the onslaught of per¬ 
vasive change, we respond in dazed, often ill-defined, and 
inadequate ways. 11 

As practitioners of the computing arts, we are thrust into 
the forefront of this technological revolution facing the con¬ 
stant realization of technological obsolescence. Ours is a 
young industry with few if any age-old standards or solutions 
to an ever increasing complexity of opportunities and chal¬ 
lenges. How are we to survive for a career that may span 
fully four decades? 

Certainly education is offered as a possible solution, but 
who is to prepare and disseminate appropriate state of the 
art courses for an increasingly diverse and specialized body 
of knowledge? Are we not facing a dilemma of enormous 
magnitude whose solution will require the resources and 
commitment of significant numbers of us? What will be the 
cohesive force to organize and coordinate our efforts? 

Has not our answer been to some extent to join together 
with peers of common interests to develop programs, ma¬ 
terials, and vehicles with which to attack such problems? In 
fact, are we in this regard much different from other indus¬ 
tries which have experienced similar needs? I believe not, 
and anticipate the increasing need for a developing, respon¬ 
sible, and viable profession. 

What exactly is a profession, or more specifically, a 
learned profession? To understand the concept, one can 
look to other so called professions for criteria. One of the 
more common is tradition. For example, the story is told of 
a doctor, engineer, and politician arguing about whose 
profession was the oldest. The doctor said there was little 
doubt that medicine was the oldest profession because hu¬ 
manity was preceded by the first medical operation—the 
removal of Adam’s rib. However, the engineer objected 
arguing that engineering preceded medicine by virtue of the 
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great project of creating the world from chaos. To this the 
politician inquired, “And who do you think created the 
chaos in the first place?’’ 9 

While data processors may not have a claim to old age 
for their profession, it might be observed that many feel we 
certainly have a legitimate claim for creating chaos. 

Perhaps some light might be shed on this matter by looking 
at the definitions of a profession and a professional. 

The simplest definition Webster offers is that a profes¬ 
sional is one “following an occupation as a means of live¬ 
lihood or for gain.” Such a definition qualifies virtually 
anyone and everyone. Somehow, though, the lack of dis- - 
tinguishing criteria diminishes the concept and leaves it less 
than useful. 

Hardly satisfied, definition two catches the eye. Profes¬ 
sional is . . . “pertaining or appropriate to a profession.” 
Further investigation reveals that a profession involves 
... “a vocation requiring knowledge of some department 
of learning or science.” Here at least are some minimal 
criteria beyond merely holding a position in the industry 
(i.e., knowledge and learning). A professional is one “en¬ 
gaged in one of the learned professions,” and is also one 
. . . “RESPECTED BY THE PROFESSION.” 

In this more lofty view of professionalism, one discerns 
other criteria pertaining to . . . “professional character, 
spirit, or methods,” and again, the . . . “standing practice, 
or methods of a professional as distinguished from an ama¬ 
teur.” 10 

Clearly there is a world of difference between the percep¬ 
tions of a professional conveyed by these later definitions 
from the extreme of mere employment. 

One might conclude then that to be professional is merely 
a matter of definition and clearly that a definition could be 
devised to cover anyone desired. However, to do so seems 
hardly productive as it serves only to camouflage any real 
differences among practitioners. What is sad is that such a 
strategy appears to be effectively implemented in the com¬ 
puting industry. At best passive, yet more often active re¬ 
sistance to any attempt to differentiate significant variables 
among computer practitioners is prevalent in the industry. 
One must ask why? 

A complete answer would have to consider many possi¬ 
bilities among which would be the desire to avoid exposure 
of the incompetent, insecure, paranoid, and even the ama¬ 
teur. However evident such motivations and the real need 
to address the problems of such people, the intent here is to 
investigate the competent practitioner to attempt to under¬ 
stand his apathy in such matters and his reluctance to iden¬ 
tify the characteristics of his own competence. Such serious 
deficiencies betray an absence of maturity among such prac¬ 
titioners. 

Maturity is a state of being; an expression of the full 
development of an individual’s ability to differentiate reality 
and delusion. If one can demonstrate a serious deficiency in 
such development of a person or groups, then one has the 
basis to challenge its maturity. 

This is precisely what is proposed here. Since the com¬ 
puting industry is seriously neglecting the critical need to 
differentiate professional development among its numbers 


in that it tolerates a simplistic blanket usage of the term 
professional that masks the incompetent and amateur in its 
midst, then in that respect it lacks maturity. It is guilty of 
ignoring the reality of differentiating criteria among practi¬ 
tioners and of deluding itself into believing no such discrim¬ 
ination is necessary. 

Not all practitioners of the computing arts are guilty of 
such lack of maturity. Many have addressed the problems, 
have worked diligently to devise sound definitions, and have 
tried to accomplish the identification of professional char¬ 
acteristics, but unfortunately many more have chosen to 
negate or compromise any achievements obtained. For the 
most part such contributors have been ostracized for their 
efforts and bullied into silence by the masses whose com¬ 
fortable but erroneous self-image would be shattered by a 
glimpse of the truth. 

Frankly, there is no such thing as a data processing profes¬ 
sional! At least, not in the sense most people in this industry 
seem to use the term. We have no clear example of a profes¬ 
sional directly comparable to a doctor, lawyer, or CPA. It 
is suggested that the term is being misused and in reality 
twisted to meet the ego needs of an immature industry. 

However, the potential of a profession exists and could 
easily be established through the efforts of significant sup¬ 
port from practitioners. What then would be required to 
qualify as a profession? 

Richard Canning, in the EDP Analyzer (December 1968) 
listed a number of criteria by which any group could judge 
if it would qualify as a profession. 9 

1. A profession has a defined body of knowledge of high 
intellectual content acquired by training-in-depth. 

2. A profession has defined standards of competence and 
certification that the professional meets those stand¬ 
ards. 

3. A profession has a code of ethical behavior. 

4. A profession has at least one professional society 
aimed at advancing the welfare of its members. 

5. A profession has responsibility to society to perform 
in a competent, ethical manner. 

6. The members of a profession generally are not under 
the control of non-professional management (with re¬ 
gard to professional decisions). 

7. The members of a profession have a loyalty to their 
profession which may transcend their loyalty to their 
employers. 

8. The members of a profession may be licensed by the 
state to practice the profession if it affects such things 
as public health, public safety, property rights, or 
schooling of the young. 

9. A profession has the right and ability to eject someone 
from the field for being incompetent or unethical. 

Now the question is . . . Does the computer field qualify? 

Certainly a body of knowledge in the computer field is 
being defined. Academic programs, educational programs of 
professional societies and an increasing body of literature 
are continually refining our understanding of minimal knowl¬ 
edge requirements. However, the technological and infor- 
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mation explosion will require diligent attention to the main¬ 
tenance of these knowledge requirements. Who is to fund 
and man these efforts? 

We have a certification program through the Institute of 
Certification of Computer Professionals (ICCP) which is 
being seriously developed, expanded, and implemented. But 
still there are few if any clearly defined standards of com¬ 
petence, although efforts are under way to attempt their 
definition. Who should be responsible for such standards 
definition? 

The ICCP has a published code of ethics as do most of 
the existing professional societies. However, most of these 
codes are “Mother and Apple Pie” documents with no en¬ 
forcement capability. Who must take on the continuing and 
sensitive task of defining ethical behavior, and more criti¬ 
cally, who will be responsible for the enforcement of such 
codes? Only ICCP has an enforcement mechanism in place 
but it has yet to be exercised. 

Most data processors are not independent decision-mak¬ 
ers, nor has there been much indication of a willingness to 
transcend their loyalty to the company and their paycheck. 
Furthermore, many argue that one ought not take respon¬ 
sibility for questioning the propriety of their management or 
employers. What reinforcement should and could be pro¬ 
vided to encourage responsible professional behavior in the 
public interest regardless of potential company reprisal? 

Notice the need for the recognition of a legitimate and 
perhaps necessary licensing role. In spite of such recognition 
by increasing numbers of responsible leaders in the industry 
as well as the private and public sectors of our society, an 
overwhelming majority of practitioners now reject the con¬ 
cept. 

Finally, no one to my knowledge has ever been ejected 
nor disciplined by any society for being incompetent or 
unethical. 

What all this means is that the computing profession, if 
there is one, is only an emerging profession and much needs 
to be done if in fact it is ever to achieve a legitimate status. 

Many practitioners argue and act as though such a situa¬ 
tion were desirable and that there is no need for concern. 
With many of the benefits of a profession emerging anyway, 
why be concerned with the legitimate recognition of the 
profession? Let things take a slow, natural course and all 
will come out in the end. In the meantime no one need be 
threatened and all can use the term “professional” as they 
please at no cost to anyone. 

Here again is a manifestation of immaturity, for the satis¬ 
faction of the ego need to be regarded as a professional is 
accomplished through the delusion that such a choice is 
within the power of the computing industry alone. Ignored 
is the reality that the public, legislatures, the courts, em¬ 
ployers, and other professions are expressing an entirely 
different point of view. While naively awaiting recognition 
of professional status, events are occurring that may well 
forever deny such status to computing personnel. 

In December of 1971, the U.S. Department of Labor’s 
Wage and Hour Division decided that programmers were 
not exempt and should not be considered professionals. 7 

This precedent supported a January 1976 decision handed 


down by a U.S. District Court judge in Tennessee that again 
ruled programmers not exempt since they did not satisfy the 
provisions of the Fair Labor Standards Act of 1938, Section 
213 which specify that “any employee employed in a bona 
fide executive, administrative, or professional capacity” is 
exempt from overtime pay. 1 

Seeking professional recognition, programmers have or¬ 
ganized professional societies, sponsored professional activ¬ 
ities, and encouraged professional development of the prac¬ 
titioner, but also all to no avail. 

What happened? Where did they slip up? First of all, in 
their lack of attention to the necessary legal definitions and 
secondly, because of divergent motivations among their 
numbers. 

Lacking a defined and recognized professional status 
sealed the fate of programmers who now face legal prece¬ 
dence in their struggle to achieve a profession. Are the rest 
of us likewise to be denied? 

Close upon the heels of this bombshell has come the 
controversy in several states during 1976-77 concerning the 
tax status of computer software. The battle continues and 
the Data Processing Management Association (DPMA) as 
reflected in an August, 1977 position statement is arguing, 
“. . . whether data processors will receive equal treatment 
under tax laws as services by other professions.” 8 This is 
an interesting argument considering the status of the so 
called computer profession, and it certainly puts the industry 
on notice to its exposure unless its professional status is 
established. 

But the question now becomes, how long will this process 
take? A corollary is how much time is available for comple¬ 
tion? 

The answer depends upon projections of current and fu¬ 
ture events now evident and believed to be inevitable. 
Among these are governmental intervention such as has 
occurred with legislation involving privacy and electronic 
funds transfer systems. 

Concerns that the public interest must be protected is 
causing legislatures to become increasingly sensitive to the 
impact of computer technology. How long are they going to 
tolerate the technology without regulation? 

If one accepts the inevitability of such regulation within 
limited time frames, the argument becomes compelling for 
accelerating formation of the profession and regulation 
within it. In fact, the passage of regulatory legislation could 
make all the prior concerns empty rhetoric. 5 

It is in the belief that such would happen, and especially 
that it would come in a relatively short time frame, that has 
prompted many of the need to address the controversy. 
However, such legislation is not without peril and inade¬ 
quate preparation must be avoided if possible. 

Such has been the appeal of this author in the past who 
has been concerned that the industry is faced with potential 
for the establishment of a profession, and that it would seem 
prudent to prepare for it. 2-6 The argument has been made 
that certain things must be done to prepare. In particular, 
identification must be made of who would be a professional, 
what he would do that would differ from non-professionals, 
and how he would perform those activities. 
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Therefore, the real issue is the formation of a legitimate / 
profession, regulated to protect the public interest, to insure 
the quality of the professional, and to provide him with the 
working tools with which he will perform. 

If it maintains a head in the sand attitude, the industry is 
likely to find itself one day regulated by government estab¬ 
lished in desperation to control a technology the public per¬ 
ceives as much too powerful, self-serving, and threatening. 

Such control may even exclude a DP profession and rel¬ 
egate it to the role of technicians employed by and regulated 
by some other profession such as accounting, or even worse, 
give rise to unionism. There are ample reasons to fear these 
eventualities. 

If the computer industry is to participate in this regulation 
of computer related areas rather than perform as technicians 
under the guidance of other professions, then we must es¬ 
tablish ourselves as acceptable professionals and in short 
order. Are we going to accept technician roles to other 
professionals? Or worse yet, unionization as technicians of 
the laboring class? 

Let it be again perfectly clear that the use of the term 
professional in our industry is merely a dream or hope for 
the future. True, many are working for the reality but the 
vast majority are either apathetic or actively opposed to the 
concept. It is this apathy and opposition which must com¬ 
mand our interest, for a lack of understanding of the motives 
of such opposition may well spell out doom to the efforts to 
legitimatize a computing profession. What then might we do 
to create a legitimate professional in our society? 

One formal effort to achieve such professional status is 
the certification movement. Here is attempted a clear and 
comprehensive articulation of peer evaluation and approval 
with attendant subscription to a continuing practice of eth¬ 
ical behavior. This, of course, is the realm of the Institute 
for Certification of Computer Professionals (ICCP). This 
effort is essential to the development of a professional group 
of people and represents the most significant industry effort 
to improve and regulate itself. At this level one finds most 
of the generally mentioned and accepted attributes of the 
definition of a profession. 

Culminating the work of many these past years, the cer¬ 
tification movement promises the industry its best potential 
of achieving a profession and the practitioner his best chance 
to aspire to a truly mature professional status. 

Full definition and development of a computing profession 
under the umbrella support of most of the significant com¬ 
puting societies is possible through the Institute for Certifi¬ 
cation of Computer Professionals (ICCP). Charged with the 
task of identifying significant discriminating characteristics 
among practitioners and certifying against such gives rise to 
hope for a profession. Definition of and attention to the 
characteristics of such a profession in addition to programs 
directed toward their achievement remain the best hope of 
the industry. However, while these efforts are often spoken 
of and are of interest, such is again not the purpose of this 
presentation. What is critical here, is the serious lack of 
support for these efforts. 

The trade press ranks with superficial and often foolish 


criticisms of the serious attempts to improve the situation 
of computer professionals. Rarely, however, rises a thought¬ 
ful defender to spoil such attacks. 

Rarely do we call our employers attention to these cre¬ 
dentials or encourage our colleagues to acquire them. Rarely 
are such credentials rewarded. Rarely do we volunteer the 
time or effort to assist those making the sacrifices in our 
behalf. How then can we truly consider ourselves mature 
professionals, if indeed we are professionals at all? 

Finally, why do we continue to tolerate those who use the 
label of professionalism itself to attack any serious attempt 
to achieve such? These people become especially agitated 
by any hint of assessment, certification, licensing, or any 
other potential measure of their credentials, yet we tolerate 
their frequent attacks of those seriously attempting to estab¬ 
lish legitimacy for that which these critics take for granted 
for themselves. Their arguments against an elite class of 
practitioners also appears weak. If being elite means being 
concerned, involved, improving ones knowledge and skills, 
and contributing to the profession, then hooray for the elites. 
The cry of “ego trip” is no more valid for the achiever than 
it is for the non-achiever. Is it not much more ego involved 
to attribute characteristics to oneself without the verification 
of peer group evaluation? 

What is suggested is that the industry begin to show ma¬ 
turity in at least three ways: First, that all assist in obtaining 
credentials reflecting clear definitions of distinguishing 
professional characteristics. Second, participate in imple¬ 
menting and acquiring legitimacy for these credentials. 
Third, cease to tolerate the distracting and demeaning at¬ 
tacks upon efforts to develop and mature the profession by 
actively responding to those who would diminish or destroy 
the efforts to improve our industry and performance within 
it. 

Surely, tradition, common sense, and responsible self-in¬ 
terest dictate the formation of such a recognized computing 
profession. 

The decision rests with us in the industry. All are urged 
to become concerned, to get involved, and to make com¬ 
mitments such that many voices and talents will be engaged 
in finding solutions to the challenges of the future toward 
the profession. 

In this spirit remember the visionary words of George 
Bernard Shaw as he wrote: 

Some men see things as they are, 
and ask why? 

I dream of things as they might be, 
and ask why not? 
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Designing and debugging careers for 
women in the computer industry 


by THELMA ESTRIN 

Brain Research Institute, University of California 
Los Angeles, California 


Women’s pursuit of careers in the computer industry is 
consistent with our national commitment to equal opportu¬ 
nity and with our national need to utilize all available talent 
in support of scientific research and industrial efforts. 

Computer science education and immediate employment 
on the first steps of the career ladder are readily available 
for women but after these initial steps women still face 
cultural, educational and institutional barriers which block 
them from advancement to the top. Women are finding, as 
men have, that superiority in technical knowledge and com¬ 
petence are not sufficient to guarantee advancement. In 
addition, women must carefully plan their careers and re¬ 
main alert to those societal attitudes, the hidden “bugs”, 
that prevent them from successfully implementing their long¬ 
term career goals. 

The concept of career planning is receiving much attention 
by professionals in all fields, but the rapid rate of techno¬ 
logical and methodological innovation in the computer in¬ 
dustry gives special urgency to the need for computer 
professionals to use planning techniques to combat technical 
obsolescence and to aid upward mobility. Career planning 
deals with the identification of ones goals, skills, and prior¬ 
ities and the designing of a career with alternative options. 
It involves deciding if one wants to pursue a technical or a 
managerial career and poses questions such as: What type 
of a job do I want in the next years? What is the knowledge 
base? What are the skills needed and at what level of com¬ 
petence and experience? What types of inter-personal rela¬ 
tionships are important? 

Many factors that relate to professional development 
apply equally to men and women but career planning is 


especially important to help compensate for the distinct 
problems women encounter with upward mobility, and for 
women who want to combine career advancement with fam¬ 
ily life. There are very few women at the top in any business 
or profession and the lack of role models serves to discour¬ 
age women who might consider working towards leadership 
status in the computer field. Qualified women are often 
reluctant to move ahead because upward mobility requires 
significant deviation from social norms. In other cases, 
women may be hindered by the attitudes of men, who are 
their peers and supervisors. Withholding promotion of 
women may not be overt, but due to unconscious discrimi¬ 
nation based on sex-role stereotyping and societal conven¬ 
tions. 

The panelist^ will discuss various approaches to career 
planning. The opening panelist, a career development con¬ 
sultant, will present a model of career stages based on ac¬ 
cepted theories of adult development. The other panelists, 
computer professionals in the areas of software, hardware 
and technical marketing, will discuss factors and strategies 
which they have found useful in advancing through career 
stages in the computer industry. They will also discuss bar¬ 
riers to women’s full and equal participation which should 
be considered in career planning. 

This panel is not for women only and men are encouraged 
to attend. In addition to benefiting from those aspects of 
career planning which are of value to all professionals, men 
can gain insights to improve their working relations with 
female colleagues from an awareness of the special problems 
professional women encounter in pursuing non-traditional 
careers. 
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A panel session—Computers in early education 

SESSION CHAIRMAN—ORLANDO S. MADRIGAL 

California State University, Chico 


Panel Members 

Alan Kay—Xerox Palo Alto Research Center 
David Moursund—University of Oregon 
John Maniotes—Purdue University 
William G. Lane—California State University 
A. M. Banks—Beyer High School 


PANEL OVERVIEW—Orlando S. Madrigal 

This session will have five panelists/speakers who will 
discuss the use of computers at the pre-college level. The 
speakers will include topics related to: 

• The cost-effectiveness of the use of computers for high 
school education, by William G. Lane—California State 
University, Chico. 

• Computer Science education for preservice elementary 
school teachers, by David Moursand—University of 
Oregon. 

• Programming for children on a personal computer. To 
include a brief movie that will demonstrate the 
speaker’s experiences with children who are taught pro¬ 
gramming at an early age. By Alan C. Kay—Xerox Palo 
Alto Research Center. 

• The teaching of micro-computers in high school, by 
Amberse Banks—Beyer High School, Modesto, Cali¬ 
fornia. 

• The state of high school data processing programs, by 
John Maniotes, Purdue University. 


SPEAKERS AND TOPICS: 

(1) Professor David Moursund, University of Oregon 
Title: Computer Science Education for Preservice 
Elementary School Teachers 
Abstract: Calculators and microcomputers are now 
cheap enough so that elementary schools can afford 
to give students quite good access. Thus this equip¬ 
ment could be playing a major role in the curriculum. 
But progress to date has been slow. A major cause is 
lack of suitably trained teachers. Preservice elemen¬ 


tary school teachers need to receive a substantial in¬ 
troduction to computer literacy, teaching using cal¬ 
culators and computers, and teaching about calculators 
and computers. The author has been involved in the 
development of courses of this sort, and is currently 
teaching such a course. This paper includes a detailed 
course outline and discussion. References to the text 
used, as well as supplementary material, are given. 

(2) Professor John Maniotes, Purdue University, Colu- 
mett Campus 

Title: The Interface Between High School Data Proc¬ 
essing Students and the Two-Year Community 
College Programs in Computer Science 
Abstract: Data and problem areas will be presented 
to explore the transition of high school students, with 
data processing background, to a formal 2-year cur¬ 
riculum in Computer Science. More and more high 
schools teach computer-related courses. Students in 
these programs are able to phase into a college pro¬ 
gram in computer science including the exemption 
from the basic courses including introductory pro¬ 
gramming and concepts. A high school data processing 
program that is well articulated with college level pro¬ 
grams will greatly benefit the students in the program. 
There will be less course duplication and students will 
be able to obtain advance placement in the Computer 
Science area. 

(3) Professor William G. Lane, California State Univer¬ 
sity, Chico 

Title: Cost-Effectiveness of the Use of Computers 
For High School Education 
Abstract: As more and more high schools start offer¬ 
ing computer programming courses, the cost of high 
school education will greatly increase. And with the 
trend towards a “no increase” in school budgets, new 
avenues of educational financing will have to be pur¬ 
sued in order to allow school districts to pursue the 
addition of basic computer science to existing high 
school curricula. It will be shown that various avenues 
of cost-sharing among computer users in a given area 
can reduce the “per student cost” for computing use 
to a level that will be within the budgetary restrictions 
of most school districts. Another area that will affect 
cost is the need for high school teachers who can 
qualify as teachers in the Computer Science area. The 
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current move to establish high school credential pro¬ 
grams in various states will be discussed. 

(4) Dr. Alan Kay, Xerox Palo Alto Research Center 
Title: Experiences With “Small Talk” and Pre- 

Schoolers 

Abstract: The author has been involved with the de¬ 
velopment of personalized computing systems includ¬ 
ing both hardware and software. A major result of his 
work is the “Small Talk” system which utilizes a mini 
computer that can be used interactively to train pre¬ 
schoolers in the use of computers. The system utilizes 
a specially designed terminal that will allow the user 
to perform interactive color graphics without prior 
training in the use of computers. Hundreds of pre¬ 
schoolers have gone through the “Small Talk” sys¬ 
tem, and interesting observations have been noted 
including the users response to a unique computer 
environment and eventually relating these experiences 
to the classroom. 

These systems, although not yet commercially 
available, can be considered as the forerunner to per¬ 
sonalized computing where users can gain exposure 
to computing at an early age. 

(5) Mr. Amberse Banks, Beyer High School, Modesto, 
California 

Title: Augmenting a High School Curriculum With 
Microcomputer Education 

Abstract: Microcomputer education can be paralleled 
to the traditional electronics training for television and 
stereo systems. However, microcomputers are more 
complex since computer programming must be in¬ 
cluded in the students work. 

The author has devoted two years to the develop¬ 
ment of high school courses which train students in 
the intricacies of building and programming micro¬ 
computers. More and more students are getting in¬ 
volved in these courses. A number of these students 
have aspirations for college-level training in Computer 
Science and eventually enter the field as computer 
professionals. 


PROGRAMMING FOR CHILDREN ON A PERSONAL 

COMPUTER—Alan Kay 

The “Small Talk” system utilizes a mini computer with 
an attached terminal that can be used interactively to train 
pre-schoolers in the use of computers. The system utilizes 
a specially designed terminal that will allow users (pre¬ 
schoolers) to perform interactive color graphics without 
prior training in the use of computers. Hundreds of pre¬ 
schoolers have gone through the “Small Talk” system, and 
interesting observations have been noted including the users 
response to a unique computer environment and eventually 
relating these experiences to classroom learning. 

These systems, although not yet commercially available, 
can be considered as the forerunner to personalized com¬ 


puting for children where users can gain exposure to com¬ 
puting at an early age. 


COMPUTER SCIENCE EDUCATION FOR 
PRESERVICE ELEMENTARY SCHOOL 
TEACHERS—David Moursund 

Calculators and microcomputers are now cheap enough 
so that elementary schools can afford to give students quite 
good access. Thus this equipment could be playing a major 
role in the curriculum. But progress to date has been slow. 
A major cause is lack of suitably trained teachers. Preservice 
elementary school teachers need to receive a substantial 
introduction to computer literacy, teaching using calculators 
and computers, and teaching about calculators and com¬ 
puters. This talk includes a detailed course outline and dis¬ 
cussion of course requirements for such training. 


THE STATE OF HIGH SCHOOL DATA PROCESSING 

PROGRAMS—John Maniotes 

In recent years, computers have had a profound effect on 
all of the institutions of society, including early (and late) 
education. If we, as educators, are to prepare students for 
the future, we need to participate fully in the computer 
evolution/revolution and to teach our students to use, un¬ 
derstand, and live with these logical machines that will exert 
such a significant influence on the rest of their lives. 

The number of computer and data processing related 
courses and programs a high school has to offer depends 
upon the size of the school, the availability of experienced 
EDP or computer science teachers, the attitude of the 
school’s administrators, and above all the amount of funds 
allocated to the EDP program. Although some schools con¬ 
tinue to overlook the impact that computers have had in the 
business and scientific world, others have found economic 
ways to incorporate some kind of computer-related courses 
in their curriculum. 

Several programs that have been categorized according to 
the equipment available to the students are described. These 
programs include the: 

• Textbook concept programs 

• Input/Output terminal programs 

• Centrally located computer concept programs 

• Home-based computer concept programs 

For those schools which lack the aforementioned pro¬ 
grams, all sorts of extracurricular activities can be found 
intermingled with the obligatory academic subjects. In par¬ 
ticular, the role of the ACM computer clubs, science and 
math clubs, and the explorer posts are described. 

Finally, some of the major problems facing high schools 
in implementing vocational data processing programs are 
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explored. In particular the problems associated with: 

• Curriculum objectives and guidelines 

• Recruitment of qualified instructors 

• On-going teacher education in EDP 

• Instructional materials 

• Computer oriented aptitude tests 

• Funding to support the EDP programs 

• Physical facilities for computer hardware 

• Computer software and application packages 

• Recent impact of microcomputers, intelligent terminals, 
and programmable calculators 

• Interfacing with 2-year BDP and 4-year CS programs at 
colleges and universities. 


COST-EFFECTIVENESS OF THE USE OF 

COMPUTERS FOR HIGH SCHOOL EDUCATION— 

William G. Lane 

A decade of program development in elementary and sec¬ 
ondary education has been characterized by oversell, under¬ 
delivery and poor timing. CAI projects, which promised to 
raise the overall class growth “X” grades, while letting 
everyone proceed at his own pace, ran into a number of 
problems: 

• Parent taxpayers, whose only probable previous expe¬ 
rience with the computer was in trying to get a credit 
card purchase error corrected, raised much concern 
over “turning the education of our children over to a 
computer.” 

• Teachers were resistant to adopting new methods that 
too frequently were perceived as having a potential for 
lowering the number of faculty positions and that had 
the effect of reducing their status to a regulated learning 
manager rather than an independent learning deliverer. 

• Curriculum personnel and school administrators were 
asked to operate in a new environment for which they 
had little or no training. 

• Marketing analysts had underestimated the cost and 
length of the development effort and over-estimated 
early market, causing computer companies to become 
disenchanted and to withdraw support. 

• Low cost hardware technology required for broad ac¬ 
ceptance had not yet been developed. 

Given these problems, it is not surprising that computers 
in the elementary secondary education areas have not even 
begun to reach their potential. But times have changed; 
hardware costs have lowered, TV games and microproces¬ 
sors have become a major factor in the home market, the 
computer is no longer perceived as a job-eating monster 
and, as such, we are now probably ready to again access 
the feasibility of installing computers at all levels of our 
school system. 

Several pilot studies are evaluated and analyzed for their 


potential: 

• The use of computer graphics in the teaching of me¬ 
chanical drawing at the junior high level. 

• The use of modularized instruction and computer-based 
testing in lower elementary levels. 

• The use of a computer in handicapped education. 

Each of the above studies showed a need for differing 
systems and peripheral architectures, as well as responsi¬ 
bility for the classroom teachers. These led to the require¬ 
ment for differing architectural concepts and approaches. 

TEACHING MICRO-COMPUTERS IN HIGH 

SCHOOL—A. M. Banks 

In view of the impact of computers upon our society, few 
people would question their inclusion in the secondary 
school curriculum. But why should it be done by using 
micro-computers? Can such curricula be of value to stu¬ 
dents? While it is true that such small systems do have more 
limitations than do larger systems, they also offer many new 
options to the high school program that only have access to 
micro-computer systems. 

Most computer courses in high schools are arranged to 
suit the type of equipment that is available to the students. 
Terminals are the most common form of computer access 
at the high school level; however, in many cases, the stu¬ 
dents are assigned as low priority users. The restrictions 
that are thus placed on high school programs curtail, pre¬ 
vent, or even discourage students. Also, the increasingly 
bleak financial situation of most school districts makes the 
acquisition of systems large enough to support more sophis¬ 
ticated time-sharing a virtual impossibility. Also, the rising 
costs of telephone lines makes the use of district-wide sys¬ 
tems more and more difficult. 

Highly flexible, expandable micro-computers now cost 
the same as terminals did a year ago. The newer self-con¬ 
tained, assembled systems cost so little that shcools can 
afford to build up a laboratory with several machines. Mul¬ 
tiple machines offer variety and protection against system- 
wide downtime. 

Can the computing work that is possible on a micro-com¬ 
puter compare with that possible on larger systems? Yes 
and no. First there is the matter of the languages that are 
available. Currently, there are several excellent BASIC in¬ 
terpreters and some reasonable assemblers available for 
each of the microprocessors used in micro-computer sys¬ 
tems. In addition, there are versions of PILOT, APL, PI/1 
and FORTH available for some chips. While the lack of 
COBOL, for example, is a handicap if we are training profes¬ 
sional programmers, these available languages are more than 
adequate for high schools. Software advances are being an¬ 
nounced every month for those who do not wish to write 
their own. This is an area where interested students can do 
advanced work on micro-computer systems but not in larger 
systems. 
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There is an additional aspect of computers that can be 
covered using micro-computers which cannot even be at¬ 
tempted with larger systems. It is possible to have students 
do hardware work, including maintenance and hardware 
configuration. It helps the student to view a computer as a 
tool to be used rather than as a shrine into whose presence 
only a privileged few are permitted. Instead, they find new 


ways and new places to put it to work. This kind of attitude 
is not generated when students use larger computers or even 
mini-computers. 

Thus, we see that micro-computers can match almost 
every service offered by a larger system to high school 
classes, and can offer additional opportunities to students 
that are just not feasible with larger systems. 



Computer education in higher education— 
status, alternatives and needs 

by JOHN W. HAMBLEN 

University of Missouri-Rolla 
Rolla, Missouri 


INTRODUCTION 

The cause of many of the problems associated with com¬ 
puter usage is the “overutilization of undereducated peo¬ 
ple.” The reason being, of course, that properly educated 
people have not been available. Bootstrapping by training 
existing personnel and pirating whatever other centers had 
trained was the only way that staff could be obtained during 
the late fifties and the early sixties. The late sixties saw the 
tremendous growth in the one and two-year programs aided 
by large infusions of federal monies via the Office of Edu¬ 
cation programs. The production of these programs crested 
around 1972 at a peak of nearly 28,000 during that year. The 
seventies might well become known as the decade of the 
recognition of the value of the college graduate to effective 
and efficient computer usage. Based upon my estimate of 
need and the present average rate of growth (about 10 per¬ 
cent) in the production of graduates of four-year programs 
it would take nearly twenty-four years to match the need 
and production. 

The main point here is that it will be some time before we 
need to worry about a “glut” in the market for four-year 
graduates in the computer related areas. At the master’s 
level, the rate of growth is about half that of the four-year 
production rate and it will take nearly twice as long to match 
production with need. At the doctorate level fifteen years 
will be required to match production with need based upon 
my estimates of need and current rates of growth in the 
doctorate production. 

All of my estimates are based upon the expectation that 
the professors in colleges and universities will seek out the 
advice of representatives of industry as to industries’ needs 
and keep the educational programs aligned with those needs. 
They cannot afford to do otherwise. 

In the following I present my estimates 7,9 of computer 
manpower based upon the best, though limited, data avail¬ 
able classified by job classifications and level of education 
which I believe will be required for replacement of current 
personnel and to fill new positions. Better data is available 
on manpower production than on utilization through the 
efforts of SREB Computer Science Project 1-3 and the Amer¬ 
ican Federation of Information Processing Societies 4,5 with 
support from the National Science Foundation. Data from 


the Bureau of Adult, Vocational, and Technical Education 
of the U. S. Office of Education 6 are also utilized. Manpower 
production is then matched against estimated needs. 

Although the need for computer manpower education is 
great, there is at least an equal need for most people to 
become “computer literates” as a minimum. This can be 
easily accomplished if all teachers and other college gradu¬ 
ates can become a little more than just “Computer Literate” 
(e.g., at least one course involving some programming). 


BACKGROUND 

During the period 1966-72 while I was Project Director for 
Computer Sciences at the Southern Regional Education 
Board (SREB), I conducted three surveys on Computers in 
U. S. Higher Education including their utilization and re¬ 
lated educational programs. These studies were supported 
by the National Science Foundation and resulted in three 
publications. 1-3 The first was published by the SREB and 
the latter two by the National Science Foundation. 

The first study was a survey using a stratified random 
sample of 739 institutions, from a population of 2219, se¬ 
lected by systematic random sampling within strata by the 
U. S. Office of Education. Six hundred sixty-nine or 92 
percent of the institutions in the sample responded. 

The base year for data collection was 1964-65 and projec¬ 
tions were requested from the institutions for 1968-69. The 
data collection instrument was designed by the Mathemati¬ 
cal Sciences Section of NSF to meet their existing program 
needs for planning. Consequently, emphasis was placed 
upon the financial aspects of college and university com¬ 
puter center operations and only limited information was 
gathered with regard to manpower training and utilization. 

The second survey was much more comprehensive and 
was sent to all institutions of higher education listed on the 
U. S. Office of Education’s Higher Education General In¬ 
formation Survey (HEGIS) file. Nineteen hundred sixty-five 
or 79 percent of the 2477 institutions responded. The data 
collection forms were designed with the advice and counsel 
of thirty or more national professional and institutional as¬ 
sociations and government agencies. The base year for re¬ 
porting data was 1966-67. 
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Extensive modifications were made in the data collection 
forms for the third survey. The base year was 1969-70. 

In 1972 I was commissioned by the SREB to do a paper 
on Computer Manpower—Supply and Demand—in SREB 
states. Using this as a starter, I prepared a similar report for 
all states. 7,9 Manpower production estimates were based 
upon the NSF supported surveys 1-3 and data from the 
USOE. 

During 1974, I sent questionnaires to 2,520 institutions of 
higher education (seminaries, conservatories and special 
schools were not included) to update the data obtained on 
degree programs in the 1969-70 survey. A 61 percent rate of 
response was obtained and the production estimates for 
1973-74 and 1974-75 were added to previous estimates. 7,9 

Another survey covering the 1976-77 year is nearing com¬ 
pletion and comparable data will be supplied at the NCC in 
Anahein for all tables in this paper. 

COMPUTER EDUCATION LEVELS AND 

ALTERNATIVES 

It is generally recognized that there is a definite need for 
some degree of computer education for all college graduates. 
This can be accomplished at several different levels. 

• Non-Credit Survey or Programming Short Course— 
This may consist of 8-16 hours of lectures and demon¬ 
strations and should be provided for faculty and staff 
on a continuing basis. Response by students is usually 
spotty. Very poor coverage can be expected from this 
approach. 

• A Unit on Computers and/or Programming Introduced 
in a Required College Course—This is the best way to 
insure a minimum introduction to computers for all 
students. Faculty can be trained for this teaching 
through short courses, provided good training outlines 
and materials are made available to them. 

• Integration of Computer Techniques as Tools in Most 
Disciplines—This part of the program requires an en¬ 
lightened faculty. It will be several years before facul¬ 
ties exist which are able to do this on a college-wide 
basis. However, there are several fields in which this 
can be accomplished now. Among them are business, 
economics, statistics, mathematics, physics, chemistry, 
engineering and the social, behavioral and biological 
sciences. If handled properly, the use of the computer 
can serve to motivate the student and to permit the 


TABLE I.—Numbers of Colleges and Universities Offering Computer 
Appreciation Courses with Enrollment 



1969-70 

1976-77 (Est.) 

No. Institutions 

152* 

300 

Enrollment 

18112 

50000 


* Fifty-six other institutions reported such a course on the books but not yet 
offered during 1969-70. 


TABLE II.—Courses and Enrollments in “Introductory Courses in Some 
Computer Language such as FORTRAN, PL/1, BASIC, ALGOL” 



1969-70 

1976-77 (Est.) 

Courses (Institutions) 

971* 

2,000 

Enrollment 

132,462 

300,000 


* Two hundred and six other courses were reported as being on the books 
but not offered yet during 1969-70. 


instructors to use more realistic examples in their in¬ 
struction. 

COMPUTER APPRECIATION COURSES 

Courses, such as Computer Appreciation, Computers in 
Society, Computers and Society, etc., have been popular 
for some time at many colleges and universities as is shown 
in Table I. 

INTRODUCTORY COURSE IN SOME COMPUTER 
LANGUAGE SUCH AS FORTRAN, PL/1, BASIC, 
ALGOL. 

These courses should consist of a minimum of two hours 
of lecture and two hours of laboratory each week. If handled 
carefully with regard to problem selection, this kind of 
course could be conducted as one multi-section course. 
However, better results can be obtained if some stratifica¬ 
tion is introduced when more than one section is needed. 
Such a course requires an instructor who has had some 
experience in programming in several computer languages. 
These courses will be required in some curricula and elected 
in others. The numbers of institutions (possible duplications) 
offering such courses and their enrollments are shown in 
Table II. 

VOCATIONAL TRAINING PROGRAM 

This is usually a one-year or two-year post high school 
curriculum. Several vocational training institutes initiated 
such curricula with federal and state assistance under the 
NDEA Title VIII program. Table III shows the estimated 
number of two-year programs offered and the number of 
degrees awarded. 7,9 Fifty-five (55) of these programs were 
in four-year and higher institutions offering the associate 
degree in data processing, computer programming, or com¬ 
puter technology. 


TABLE III.—Estimated Number of Two-Year Degree Programs and 
Number of Degrees Awarded 



1966-67 

1969-70 

1973-74 

1976-77 

No. Programs 

178 

405 

338 

400 

No. Degrees Awarded 

1281 

6051 

4831 

7000 
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TABLE IV.—Estimated Number of Bachelor’s Degree Programs and 
Degrees Awarded 



1966-67 

1969-70 

1973-74 

1976-77 

No. Programs 

90 

198 

302 

400 

No. Degrees 

594 

2208 

5951 

9000 


FOUR-YEAR DEGREE PROGRAM FOR COMPUTER 

PROFESSIONALS 

Although there is now a recognized need for many majors 
in computer science, information systems, etc., not many 
colleges can afford to properly staff such a program for 
several years to come. For one thing, the competition is 
very keen for new Ph.D.’s in Computer Science and very 
few exist in Information Systems. 

One hundred thirteen bachelor’s degree programs in Com¬ 
puter Science were reported in operation with 8000 majors 
and 1537 graduates for 1970-71. In addition, another 69 bach¬ 
elor’s programs were reported going with nearly 7000 en- 
rollees and 1000 graduates. The latter included 28 programs 
in Data Processing, Information Sciences, and Computer 
Science as options in various academic areas, with the most 
being in mathematics and electrical engineering. All 28 of 
these programs were offered in institutions granting a mas¬ 
ter’s degree or higher. Table IV gives estimates 7,9 of the 
numbers of bachelor’s degree programs and the number of 
degrees awarded. 

MASTER’S DEGREE PROGRAMS 

A perennial debate is whether the M.S. degree should be 
professional or research oriented. Where possible those who 
want to go on for a doctorate should do a thesis option 
otherwise little distinction should be made. The number of 
M.S. programs and degrees continue to grow as can be seen 
in Table V. 

DOCTORATE PROGRAMS 

Conte and Taulbee 8 are gathering information annually on 
the “Production and Employment of Ph.D.’s in Computer 
Science” as a Subcommittee on Manpower of the Computer 
Science Board. Prior to 1976 their data was collected only 
from Ph.D. granting departments of Computer Science, In¬ 
formation Science, etc. Sixty departments reporting for 1976 


TABLE V.—Estimated Number of Master’s Programs and Degrees 
Awarded 



1966-67 

1969-70 

1973-74 

1976-77 

No. Programs 

98 

155 

157* 

200 

No. Degrees 

742 

1442 

2854 

4000 


TABLE VI.—Estimated Number of Ph.D. Programs and Degrees Awarded 



1966-67 

1969-70 

1973-74 

1976-77 

No. Programs* 

58 

98 

90 

100 

No. Degrees 

174 

307 

369 

500 


* Includes options in Math, Electrical Engineering, etc. 


graduated 246 Ph.D.’s and estimated that 310 would com¬ 
plete the degree in 1977. Considering the number of depart¬ 
ments reporting these numbers are well in line with my 
estimates given in Table VI. 


MANPOWER NEEDS 

In References 7 and 9, I present a model which I devel¬ 
oped to produce estimates of computer manpower needs. 
The results are given in Tables VII and VIII. 

The post-secondary vocational institutions’ production 
figures are added to the associate degrees to obtain the final 
“post-secondary and two-year” estimates. 

These estimates say that we are close to producing enough 
two-year/post-secondary graduates in the computer field, 
but only one-half of the needed doctorates, one-eighth the 
number of bachelor’s and one-eleventh the number of mas¬ 
ter’s levels needed. This reinforces my earlier statement that 
we are overutilizing undereducated people in the computer 
related fields. Others may wish to take exception to the 
assumptions which led to these results. If so, I hope that I 
have made the procedures transparent enough in Reference 
7 that they can produce their own estimates. Even if my 
estimates are off by 50 percent, we still would have severe 
shortages indicated at the four-year and master’s levels for 
some years. 

The master’s degree, quite common in the management 
analysts and systems programmers categories, and not too 
uncommon among applications programmers. Some top su¬ 
pervisory positions in operations attract M.S. graduates in 
the computer and information sciences. 

Table VII gives my estimate of the distribution of com¬ 
puter staff by level of education that I believe to be desired 
for replacement if an adequate supply of manpower is avail¬ 
able at these levels. During the past years, companies have 
had to take what staff they could get and train or retrain 
them. This is an expensive route and therefore, I believe 
that given a choice, industry would prefer to hire trained 
personnel. In terms of formal education, I believe that we 
are overutilizing undereducated people in the computer re¬ 
lated occupations. 


COMPARISON WITH BLS ESTIMATES AND 
PROJECTIONS FOR 1980 

In the 1972-73 Occupational Outlook Handbook published 
by the Bureau of Labor Statistics, it was reported that there 
were over 100,000 systems analysts in 1970, over 200,000 


* 192 for 1975-76. 
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TABLE VII.—Estimated Distribution of 1973 Computer Staff by Level of Education Required or Desired for Replacements 



PS VOC and 
2-year 

4-year 

M.S. 

Ph.D. 

Total 

Percent of 
Total 

Teaching 

200 

1,500 

2,000 

2,200 

5,900 

.6 

Management 

3,000 

37,000 

60,000 

5,000 

105,000 

11 

Analysts 

1,000 

62,000 

100,000 

2,000 

175,000 

18 

Systems Programmers 

2,000 

33,000 

70,000 

— 

105,000 

11 

Applications Programmers 

10,000 

200,000 

35,000 

— 

245,000 

25 

Operators 

100,000 

100,000 

10,000 

— 

210,000 

22 

Other Operating Personnel 

100,000 

12,000 

— 

— 

112,000 

12 

Total 

Percent of Total 

216,200 

23 

445,500 

46 

287,000 

30 

9,200 

1 

957,900 



programmers and over 200,000 operators. Assuming a 10 
percent annual growth factor (Gilchrist, Computerworld, 
McGovern of International Data Corporation, and U.S. De¬ 
partment of Labor in the U.S. Economy in 1980, Bulletin 
1673, p. 47) we can expect these numbers to double by 1980. 
This extrapolation would give 200,000 systems analysts, 
400,000 programmers and 400,000 operators for 1980. How¬ 
ever, the 1970 estimate for systems analysts does not agree 
with the 1968 figures given in the U.S. Economy in 1980, p. 
59, which gives 150,000 systems analysts for 1968 or when 
extrapolated to 1970, 180,000 systems analysts for 1970 and 
240,000 for 1973, as compared to my estimate of 175,000. 
The projections for 1980 are presented on the same page 
and are 425,000 for systems analysts, 400,000 for program¬ 
mers and 400,000 for operators. Some of the differences 
result from definitions. I would guess that some of what I 
call systems programmers are counted by BLS as program¬ 
mers and others as systems analysts since no such category 
as systems programmers appears in BLS publications re¬ 
ferred to above. It is of interest to note (p. 59, also) that 
systems analysts, programmers and computer operators 
were expected to be the fastest growing occupations in the 
1970’s. 

Management personnel are most likely to be counted 
under “Managers” in the BLS classifications and on page 
293 of the 1972-73 Handbook there is an indication that what 
I call other operating personnel (excluding card punch op¬ 
erators) are included under Office Machine Operators. In 


TABLE VIII.—Future Computer Manpower Needs Per 1000 Estimated 
1973 Personnel 




Education Level 



PS Voc 
and 2-yr. 

4-yr. 

M.S. 

Doctorate 

Cause: 

Replacement: 

Deaths 

5 

5 

2 

2 

Retirements 

17 

17 

17 

17 

Growth 

100 

100 

100 

100 


— 

— 

— 

— 

Total Rate 

122 

122 

119 

119 

Total New Openings 

26,376 

55,571 

34,153 

1,095 

Estimated 1975 Productions 

19,286 

6,9% 

3,236 

493 


particular, operators of tabulating machines and related 
equipment, where such equipment is an integral part of a 
computer installation, would fall under my classification of 
“other operating personnel.” 

In BLS Bulletin 1673 referred to above it was estimated 
that there would be 120,000 computers in use by 1980 and 
that the number of process computers would reach about 
17,000. I believe that the 120,000 figure is conservative, 
depending upon one’s definition of a computer. In the same 
publication, it was estimated that for the period 1968-80 
there would be annual openings for 23,000 programmers, 
27,000 systems analysts and 20,400 operators, or a total of 
70,400 in just these three classifications. Since these three 
categories account for about two-thirds of my total esti¬ 
mates, extrapolation of the 70,400 figure gives about 105,000 
annual openings per year. This number compares very well 
under the circumstances with my estimated need of 117,000 
per year. 


COMPARISON WITH HEGIS DATA 

Gilchrist and Weber 5 refer to classification differences 
which exist between the HEGIS data and the data which I 
collected at SREB. The main reason for differences in num¬ 
ber of graduates reported is that HEGIS data, in general, 
reflect graduates of so-named departments only. For ex¬ 
ample, graduates of mathematics departments with com¬ 
puter science options would be classified as mathematics 
(or a subheading) and not computer science. The same 
would, in general, be true of graduates of other departments 
(EE, IE, etc.) with computer science options. Likewise, 
graduates of departments in business with options in systems 
analysis would not be reported under systems analysis. For 
this reason, the data reported in the inventory conducted by 
me at SREB and my updates for 1973-74 and 1974-75 should 
be considered to reflect more closely the actual numbers of 
graduates being produced in these computer related fields. 

SUMMARY 

Large numbers of personnel have been able to enter the 
computer manpower pool in the past with little training. The 
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proprietary schools had a “heyday.” Only the best and 
largest have survived. Large amounts of federal and state 
funds were poured into the vocational and two-year pro¬ 
grams. I question the value of those operated at the second¬ 
ary level and have indicated 7,9,10 that there should be a weed¬ 
ing out of the weakest post-secondary vocational and two- 
year programs. I differ with Gilchrist and Weber 4,5 in that 
I do not think that secondary schools will be significant 
producers of computer manpower. On the other hand, com¬ 
puter education at the secondary level should not be ne¬ 
glected. Secondary schools should provide enough computer 
education so that the student can learn to live comfortably 
in a computer assisted environment and also enough so that 
students pursuing post-secondary education can judge 
whether or not they wish to pursue a program of study 
which will prepare them for entry into the computer man¬ 
power pool. 
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INTRODUCTION 

The introduction of the digital computer into the academic 
environment created a situation that would lead to impact 
on how teaching and research are conducted at virtually all 
levels, and in virtually all disciplines. The further combined 
evolution of semiconductor technology with software prac¬ 
tice and methodology has resulted in a new field which has 
developed at an astounding rate. The impact of this rapid 
change in the academic environment has forced universities 
to modify their classic attitude of slow change on curricula 
and research matters to yield to external demands for edu¬ 
cation and research in computer related topics. 

Though the implication of the computer has been present 
in all areas of the academic institution, this paper will be 
directed to education about the computer; that is, computer 
science and computer engineering. Within these areas the 
changes that have occurred can in some way be observed 
by the changes that have occurred in pedagogical hardware. 
Initially, most programs in these areas relied on a “large” 
central computer resource of the institution, and perhaps, 
a modest logic design laboratory. With their introduction, 
minicomputers were acquired in increasing numbers and it 
became common for a major department to have its own 
system. More recently, low priced microprocessor systems 
are placing significant computer power in the hands of al¬ 
most any department. 

An equally significant development went on in the class¬ 
room as in the laboratory. Initially, programming classes 
were presented often by the computer center staff. These 
quickly moved into more academic settings within mathe¬ 
matics and electrical engineering departments. From this 
point options and minors developed. These, in turn, devel¬ 
oped into degree offering departments, which included in 
their offerings the tool courses but also moved up to more 
abstract courses, such as switching and automata theory, 
computability and formal languages. 

* On leave from the University of South Florida, Tampa, Fla. 


In the sections that follow, the development of the aca¬ 
demic concepts of this area of education will be considered, 
with primary emphasis on the more recent developments. 

THE DEVELOPMENT OF CURRICULUM IN 

COMPUTER SCIENCE 

The single most significant document in computer science 
education was the report of the Curriculum Committee on 
Computer Science (C 3 S) of the Association for Computing 
Machinery, most commonly known as “Curriculum ’68.” x 
The report gives detailed specifications for twenty-two 
courses fundamental to the study of computer science, as 
well as putting these courses into a prerequisite structure 
that also involves mathematics. 

While the questions of definition and justification of com¬ 
puter science are not addressed in the report, the subject 
matter is classified into three areas of computer science and 
two related areas. A core program is defined, and based on 
this core area the undergraduate program is detailed involv¬ 
ing computer science courses, mathematics courses, and 
technical electives. 

Graduate degree programs are mentioned but not detailed 
to the extent of the undergraduate program. The master’s 
program is outlined in terms of areas while doctoral pro¬ 
grams are only mentioned in general terms, with ideas put 
forth for further work. Additional topics which received 
some attention in the report included service courses and 
minors, continuing education programs, program implemen¬ 
tation problems, and facilities. 

“Curriculum ’68” may be seen as an expansion and re¬ 
finement of the 1965 report of C 3 S 2 . This earlier report spent 
a great deal more time on the definition of computer science 
as a discipline, and its relationsnip to other disciplines, es¬ 
pecially mathematics. 

The 1965 report of C 3 S was, in turn, an outgrowth of a 
panel presentation at the 1963 ACM national meeting that 
was reported in 1964 in The Communications of the ACM 
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by Perlis, 3 Arden, 4 Forsythe, 5 Korfhage, 6 Gom, 7 Muller, 8 
Keenan, 9 and Atchison and Hamblen. 10 In these reports 
one finds a definition and specification of computer science; 
courses and program description emphasizing the first 
course, numerical analysis, logic and logic design; and de¬ 
scriptions on programs then existing in the United States. 
Within the courses and program descriptions emphasis was 
placed on the role of programming projects and exercises as 
an integral portion of the programs, and the close ties of 
computer science education to mathematics, at that time, 
was noted. 


COMPUTER ENGINEERING 

While the more theoretical aspects of computing machin¬ 
ery was being addressed by C 3 S, the improvement in eco¬ 
nomics in the computer industry opened new areas in logic 
design. Not only was computer design the main objective of 
studying combinatorial and sequential circuits, but also 
new areas in process control, data communications, systems 
analysis and simulation required this knowledge. A trend 
was initiated in colleges of engineering and particularly 
among electrical engineering departments to create the dis¬ 
cipline of Computer Engineering. 

In September 1967 the COSINE Committee of the Com¬ 
mission on Engineering Education published recommenda¬ 
tions for an undergraduate course program. 11 Though origi¬ 
nally designed as an enhancement of an electrical 
engineering curriculum this report was, in effect, the initial 
development of the area of computer engineering. 

The report presents a series of subject areas in a computer 
science program in electrical engineering consisting of basic 
subject areas including programming principles, computa¬ 
tion structures, discrete mathematics, and machines, lan¬ 
guages and algorithms; and recommended elective areas in¬ 
cluding digital devices and circuits, switching theory and 
logical design, programming systems, numerical methods, 
optimization techniques, circuit and system theory, infor¬ 
mation theory and coding, functional analysis, combinato¬ 
rics and applications, probability and statistics and symbol 
manipulation and heuristic programming. 

The COSINE Committee did not regard its mission as the 
same as C 3 S and did not equate electrical engineering and 
computer science. Nonetheless, it is of note that much of 
the material recommended parallels that of “Curriculum 
’68.” The main differences appear in the advanced areas, 
and in a lack of a course or area corresponding directly to 
compiler construction. The core of the two programs are 
virtually the same, however, the work reported on by the 
COSINE Committee does not place the area of data struc¬ 
tures in the fundamental position given by C 3 S. 

The COSINE Committee issued eight other reports deal¬ 
ing with the first course in electrical engineering, 12 computer 
organization, 13 digital subsystems, 14 industries’ reaction, 15 a 
computer engineering option, 16 digital system laboratory 
courses, 17 operating systems, 18 and the minicomputer in lab¬ 
oratory programs. 19 Perhaps the lack of a stable operating 
organizational base such as a professional society prevented 


the COSINE reports from having the broad impact of the 
more integrated report of C 3 S. An effort following in the 
steps of the COSINE Committee has been the DISE Com¬ 
mittee, funded by the National Science Foundation, and 
with some of the same problems. It should be noted, how¬ 
ever, that the efforts of the DISE project are more directed 
at dissemination of materials than at the definition of pro¬ 
grams. 

CURRICULUM DEVELOPMENT SINCE 

“CURRICULUM ’68“ AND THE COSINE REPORT 

Subsequent to the publication of “Curriculum ’68” and 
the COSINE Report, considerable work has gone on relating 
to computer science education. A significant development 
in this period was the formation of the ACM Special Interest 
Group on Computer Science Education (SIGCSE), which 
provides a continuing organization for the presentation and 
exchange of ideas in the field. 

C 3 S has continued its activities following the publication 
of “Curriculum ’68.” Under the sponsorship of this Com¬ 
mittee, a series on doctoral programs was published, a sum¬ 
mer institute program for smaller colleges was conducted, 
course recommendations for smaller schools were pre¬ 
pared, 20 and guidelines for masters programs were pre¬ 
pared. 21 

Additional curriculum work has been conducted within 
ACM by the Curriculum Committee on Computer Education 
for Management (C 3 EM). This Committee has prepared 
guidelines for both graduate programs 22 and undergraduate 
programs 23 in information systems which integrates mate¬ 
rials associated with computer science with materials asso¬ 
ciated with business. 

As was indicated in the previous section, the COSINE 
Committee was active in this period, and in addition to the 
publication cited, the Committee also conducted a series of 
workshops to assist in the interpretation of their work. 

The Committee on the Undergraduate Program in Math¬ 
ematics (CUPM), also published recommendations in the 
area of computational mathematics 24 during the period. This 
work represented an extension of earlier work describing a 
mathematics program with work in computing. 25 

In addition to these efforts, the literature of computer 
science education contains numerous references to specific 
courses and topics providing an expansion of the course 
material given in the curriculum studies, and presenting 
something of a dynamic updating of the recommendations. 
The more recent work of this type emphasizes the area of 
software engineering and has been addressed in a recent 
volume. 26 A comprehensive survey of the post “Curriculum 
’68” literature in computer science education has been pub¬ 
lished, 27 and thus will not be further addressed at this time. 

NEW CURRICULUM RECOMMENDATIONS IN 

COMPUTER SCIENCE AND ENGINEERING 

The Education Committee of the IEEE Computer Society 
has been meeting regularly for over two years to design a 
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curriculum which may serve as a model for both the com¬ 
puter science and computer engineering approaches. This 
work led to the publication of a report of the Model Curricula 
Subcommittee of the Education Committee in February 
1977, 28 although a preliminary report on the work had ap¬ 
peared earlier. 29 

Two aspects of this report should be taken into account 
for its correct interpretation: 

1. The total curriculum as presented probably exceeds 
the resources or time allocations of most undergraduate 
programs and, as a result, needs to be adapted to re¬ 
gional needs, local resources, and departmental ori¬ 
entation. 

2. A central “core” of subject matter is outlined that is 
considered the kernel of an undergraduate program in 
computer science and engineering. 

There is no pretention that the report is either exhaustive, 
complete or definitive. It is claimed that it does represent 
the views of representative segments of the educational 
community acting in a committee mode with suitable com¬ 
promises. 


The general distribution of the recommended course mod¬ 
ules is divided into five areas as follows: 

(a) Digital Logic—5 courses 

(b) Computer Organization and Architecture—5 courses 

(c) Software Engineering—8 courses 

(d) Theory of Computing—4 courses 

(e) Laboratories—7 courses 

The titles of these courses are given in Figure 1. For catalog 
descriptions and more detailed information, the interested 
reader is referred to the full report referenced above. 

It should be noted that from the number of courses, the 
software engineering area is the predominant area in the 
curriculum; thus, giving the curriculum a pragmatic and 
applied flavor. In the theory of computing area, a modest 
amount of course work is recommended which presents a 
number of very abstract principles in an introductory fash¬ 
ion. 

Of particular note is the laboratory sequence. Most lab¬ 
oratories in present computer science and engineering pro¬ 
grams, whether they are a part of a given course or a sep¬ 
arate course, deal with a specific subset of hardware or 
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Figure 1—The courses of the IEEE/Computer Society curriculum recommendations 
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software problems. Therefore, a laboratory sequence should 
allow a student to become familiar with current technology 
to apply theory and to solve problems. Thus, the main ob¬ 
jective is to expose the student to methods applicable to the 
“real world,” to enhance creativity, and to develop problem 
solving skills. In the process, the student needs to learn a 
variety of techniques for designing, implementing, debug¬ 
ging, and maintaining a project. 

The approach suggested here has a two-dimension orga¬ 
nization. 

1. Vertically, the labs are organized in a series of graded 
courses in which the student advances gradually from 
well-defined experiments to more sophisticated proj¬ 
ects. 

2. Horizontally, the labs are used as media to apply hard¬ 
ware and software principles in an integrated form. 

The value of such experience becomes clear when the 
student realizes that the solution of a hardware design prob¬ 
lem requires considering its software implications and vice 
versa. 

The prerequisite structure for the program is given in 
Figure 2 which indicates the positioning of both the courses 
and the laboratories. 


THE NEW RECOMMENDATIONS OF THE 
CURRICULUM COMMITTEE ON COMPUTER 
SCIENCE (C 3 S) OF THE ACM 

During the same period that the Computer Society was 
developing recommendations, C 3 S was preparing an update 
and modification to “Curriculum ’68.” This appeared in 
draft form in June, 1977. 30 Though it will not be covered in 
this paper, it should also be noted that a subcommittee of 
C 3 S prepared draft recommendations for a program at the 
two year level at the same time. 31 These reports were pub¬ 
lished in this way as working papers to obtain maximum 
input regarding their content prior to publication of the final 
committee reports which is anticipated in 1978. 

The C 3 S report is broken down into the general areas of 
the core curriculum, Computer Science electives, the un¬ 
dergraduate program, service courses and other considera¬ 
tions including facilities, staff and articulation. 

The core material is considered to be at both the elemen¬ 
tary and intermediate level. The material at the elementary 
level is first presented as a list of topics in the areas of 
programming, software organization, hardware organization 
and data structures and file processing. It is anticipated that 
many implementations are possible from this listing of top¬ 
ics, and it is emphasized that programming projects should 
be stressed throughout the course sequence. As a guide to 
implementation, but certainly not the only implementation, 
a five course sequence is proposed at this level: CSl-Com- 
puter Programming I, CS2-Computer Programming II, CS3- 
Assembly Language Programming, CS4-Introduction to 
Computer Organization, and CS5-Introduction to File Proc¬ 
essing. Three intermediate level courses are then proposed 


to complete the core or that material required to provide 
minimum requirements common to all computer science un¬ 
dergraduate programs. They are CS6-Operating Systems and 
Computer Architecture I, CS7-Data Structures and Algo¬ 
rithm Analysis, and CS8-Organization of Programming Lan¬ 
guages. 

Following the core material illustrated in Figure 3, two 
intermediate level electives are proposed, as are nine ad¬ 
vanced courses, and provisions for special topics courses. 
The course prerequisite structure in the report is given in 
Figure 3, and the interested reader is referred to the full 
report for more detailed information. 

It should be particularly noted that the intermediate level 
core courses are recommended to contain a strong emphasis 
of fundamental concepts exemplified by various types of 
programming languages, architecture and operating systems, 
and data structures. Neither theoretical treatments nor case 
study approaches in and of themselves are adequate or ap¬ 
propriate at this level. Advanced level courses may be used 
for such treatments. Special topics courses are suggested 
and it should also be noted that which of the special topics 
are offered will be dependent on the resources of the de¬ 
partment, however, it certainly is the case that in time, some 
of the material listed under this category, should be inte¬ 
grated into the courses more fully specified, or replace entire 
courses in the curriculum. Monitoring this phase of the pro¬ 
gram will be a continuing activity of C 3 S. 

Tied into the prerequisite structure of the program are six 
mathematics courses: 

MA 1 Introductory Calculus 
MA 2 Mathematical Analysis I 
MA 3 Linear Algebra 
MA 4 Discrete Structures 
MA 5 Mathematical Analysis II 
MA 6 Probability and Statistics 

Descriptions of the courses MAI, MA2, MA3, MA5 and 
MA6 may be found in the 1965 CUPM recommendations. 32 
MA4 represents a more advanced course in discrete struc¬ 
tures than that given in “Curriculum ’68,” and builds on 
concepts developed in the study of calculus and linear al¬ 
gebra, emphasizing applications of discrete structures to 
computer science. The complete prerequisite structure is 
given in Figure 4 which includes all the courses mentioned, 
except CS 10, Computer and Society, and the Special Topics 
courses whose prerequisites will vary depending on circum¬ 
stances. 

The major will normally consist of the eight courses of the 
core material plus four additional courses selected from the 
recommended computer science intermediate and advanced 
electives with no more than two from any one specific sub¬ 
field of the discipline. It is further recommended that the 
student take mathematics courses MAI, MA2, MA3, and 
MA4, and depending on computer science electives se¬ 
lected, MA5 and MA6 may be required. 

As has been indicated, further details are available to the 
reader in the report, and in fact, this paper has only ad¬ 
dressed those portions of the report dealing with the under- 
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Figure 2—The prerequisite structure of the IEEE/Computer Society 
laboratory sequence 
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Figure 3—The core courses of the C 3 S recommended curriculum 


graduate major. It should be noted, at this point, that the 
minimum requirement for the major, as specified, requires 
48 semester hours which is certainly less than half the typical 
undergraduate program. 

CONCLUSIONS, OUTLOOK AND PROJECTIONS 

As has been shown in this paper, computer science and 
computer engineering have grown considerably in the past 
ten years. It would appear significant that it took ten years 
from the publication of the COSINE recommendations in 
1967 to the publication of the model curricula of the Com¬ 
puter Society, while it also took ten years from the publi¬ 
cation of “Curriculum ’68” to the final version of the report 
updating that document. 

What is perhaps most significant, however, is the conver¬ 
gence that can be seen occurring between the groups. 
Though working almost in total independence of each other, 


the two committees prepared guidelines remarkably similar 
in philosophy and content. 

This similarity may be best seen in considering the core 
material of the two programs; CS1 Computer Programming 
I is similar to SE-1 Introduction to Computing, CS2 Com¬ 
puter Programming II is similar to SE-2 Data Structures 1, 
CS3 Assembly Language Programming and CS4 Introduc¬ 
tion to Computer Organization cover much the same mate¬ 
rial as CO-1 Introduction to Computer Organizations, CS5 
Introduction to File Processing is similar to SE-3 Data Struc¬ 
tures II, CS6 Operating Systems and Computer Architecture 
I is similar to SE-6 Operating Systems and Computer Ar¬ 
chitecture 1, CS7 Data Structures and Algorithm Analysis 
covers much the same material as TC-2 Design and Analysis 
of Algorithms and SE-5 Data Base Systems, and CS8 Or¬ 
ganization of Programming Languages is similar to SE-4 
Programming Languages. 

As one would expect, there are differences in the rec¬ 
ommendations. The recommended core of the Computer 
Society program more strongly emphasizes the hardware 
and engineering areas, while those of C 3 S more strongly 
emphasize software and theoretical areas. But while these 
differences exist, the similarities strongly point to the fact 
that the discipline of computer science and engineering is 
approaching a well defined state. 

It is fortunate that the two committees produced their 
reports at essentially the same time. This gives the educator 
the opportunity to study both documents, and select those 
parts that best fit his institutional needs in constructing a 



Figure 4 —Prerequisite structure of the C 3 S curriculum 
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program, for it must be emphasized that these are only 
guidelines. 

Coursework may be outlined on paper, however, more 
critical questions of feasibility of implementation in a variety 
of settings exists. While it is not anticipated that there will 
be complete agreement on courses and curriculum design 
on the part of all interested parties, by continued interchange 
of ideas, at least a consensus of the critical issues to be 
covered may take place. To foster this interchange, both 
committees welcome input from all interested parties. 
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The status of computer education 
in the community and junior colleges— 
Needs and alternatives 

by JOYCE CURRIE LITTLE 

Community College of Baltimore 
Baltimore, Maryland 


INTRODUCTION 

The community and junior college is usually a multi-pur¬ 
pose, comprehensive, open-door institution, offering tradi¬ 
tional transfer programs, career programs, continuing edu¬ 
cation courses, and community service courses. Each col¬ 
lege is usually funded by both state and local taxes, and 
serves the citizens of a local geographic political division or 
sub-division. They offer access to higher education by 
means of generally lower tuition and by being within com¬ 
muting distance of the citizens they serve. They are not 
often residential campuses, generally having a clientele who 
work part-time and maintain their role in their regular family 
situation. 1,2 

Although the first public community and junior college 
was established in 1901, it was not until the 1940’s that the 
President’s Commission on Higher Education called for 
making education more generally available through the es¬ 
tablishment of two year colleges. Later commissions, in 
1960 and 1970, recommended that a college be within com¬ 
muting distance of almost every citizen of the United States. 
By serving as the “open-door” of opportunity to those who 
had no chance for higher education before, the two-year 
institutions have been credited as the “social leveler” in our 
society, and have brought this country closer to the goal of 
universal higher education. 2 


COMPUTER EDUCATION 

The earliest programs in the computer field in the two- 
year colleges were those offered in California in the late 
1950’s. These early programs were generally built around 
unit record equipment and computers acquired through sub¬ 
stantial discounts from IBM, in conjunction with support 
from the federal government. Because business data proc¬ 
essing is the name generally used for record-keeping activ¬ 
ities of a business or industry, most of the early programs 
carried this name and were offered through the auspices of 
the Business or Business Administration department. Many 
of these programs still exist, and often are named Data 


Processing Technology, Information Systems, Computer 
Programming, or Computer Science. 

A typical career program in a two-year college offering an 
Associate in Arts or Sciences includes anywhere from 62 to 
70 semester credits of course work. This generally includes 
24 credits of general education work, including English, 
Mathematics, and Social Studies, 12 to 15 credits in subjects 
in Business, and 18 to 24 credits of computer courses. Most 
of these programs generally include: three credits of an 
introductory data processing, three credits of an introduc¬ 
tory computer programming language, as many as eight 
credits of COBOL programming, perhaps three credits of 
RPG programming, and six credits of systems analysis and 
design. Students sometimes have an opportunity to take one 
course in operations, and in some instances are included in 
an internship or work-study program. The mathematics re¬ 
quirement is generally either Business Mathematics or Gen¬ 
eral College Mathematics, sometimes including a course in 
Elementary Statistics. 

Additions to the offerings of the two-year colleges in¬ 
cluded, in many instances, a “scientific” option. These pro¬ 
grams emphasize the use of the computer in the program¬ 
ming of statistical and data-gathering applications, in 
forecasting, and in mathematical and engineering work. 
Many of these programs are offered under the auspices of 
the mathematics departments; some of them are called com¬ 
puter science. More recently, two-year colleges near four- 
year programs have initiated the first two years of a transfer 
program, also generally entitled computer science. The sci¬ 
entific option generally differs from the business data proc¬ 
essing or commercial option in these ways: classical math¬ 
ematics up through the calculus is substituted for the courses 
in business, statistics is required, emphasis in computer 
language is on FORTRAN, and assembler programming is 
stressed in more depth. The transfer program, however, 
generally offers a course in data structures, one in computer 
organization or architecture, and a course in numerical 
methods. 

A few schools offer an option within the Engineering or 
Electronics Departments. Students learn fundamentals in 
digital and analog computer logic and circuitry, and often 
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are employed by the manufacturers of computers to be fur¬ 
ther trained as customer service engineers. 

Certificate programs are offered in some two-year 
schools. These are generally one year in length, but some¬ 
times are one semester. Most computer operations certifi¬ 
cate programs are one year, while certificate programs in 
data entry are generally shorter. In many states, these types 
of programs are offered by the secondary vocational-tech¬ 
nical schools rather than in the community and junior col¬ 
leges. 

Many of the community and junior colleges offer courses 
about computers as a service to other departments and pro¬ 
grams. Students majoring in Business or Accounting gen¬ 
erally take one introductory course, as do students in med¬ 
ical records and paraprofessional health programs. Some 
schools offer “professional development” courses for in¬ 
dustry, although the popularity of these courses has de¬ 
creased at the community and junior college level as states 
have begun to limit the number of credits that may be trans¬ 
ferred from these institutions to upper division colleges. 
Community service courses for no credit or for continuing 
education units (CEU’s) remain popular with local com¬ 
munity groups, especially within off-campus centers in local 
industry. 

CURRICULUM GUIDELINES 

Several curriculum guidelines for two-year programs have 
been prepared by the U.S. Department of Health, Education 
and Welfare Office of Education. A business data processing 
curriculum was prepared and released in 1963, 3 covering a 
broad range of careers in the computer field. In 1964, a 
report for electronic data processing in engineering, science, 
and business 4 gave suggested techniques for use in courses 
of study. A report on scientific data processing technology 
was released in 1970, 5 and in 1973, the earlier 1963 report 
was revised. 6 The 1963 publication was perhaps the most 
important influence on the establishment of programs during 
the 1960’s. 

In 1969, the American Association for Community and 
Junior Colleges (AACJC), sponsored the publication of two 
reports, with the assistance of a grant from Hewlitt-Packard 
Corporation. One concerned curriculum; 7 the other con¬ 
cerned college-wide computer usage. 8 There are currently 
no plans to update these reports by AACJC. 9 

A report giving recommendations and guidelines for a 
career program in computer programming for community 
and junior colleges was recently released by the Subcom¬ 
mittee for Community and Junior College Curriculum of the 
Association for Computing Machinery (ACM). 10 This report 
is based on the results of extensive discussion held between 
June 1975 and February 1977 with community college edu¬ 
cators, industry representatives, and professional society 
representatives. The report appeared as a working paper in 
the SIGCSE Bulletin, along with a working paper for an 
undergraduate program in computer science, prepared by 
the Committee on Curriculum in Computer Science of the 
ACM, to receive reactions and comments before final pub¬ 
lication. 


This report represents the first set of curriculum recom¬ 
mendations to be produced for the community and junior 
college level by a professional society. In earlier ACM cur¬ 
riculum reports, however, providing educational programs 
for applications programmers is credited to the community 
and junior colleges. 11,12 The 1977 report is intended for the 
preparation of entry-level or trainee computer programmers 
who will work in an applications setting to support the gen¬ 
eral, administrative, and organizational functions of indus¬ 
try, commerce, business, and government service. Even 
though it is designed to prepare students for jobs, it empha¬ 
sizes the need for a sufficient foundation for continued learn¬ 
ing and advancement in the field. It is hoped that this report 
will encourage the re-evaluation of existing programs as well 
as serving as a guideline for the creation of new programs. 

The ACM Community and Junior College Curriculum Re¬ 
port gives 14 objectives of the program, covering technical 
and non-technical skills. Instead of providing one sample or 
recommended curriculum structure, if offers guidelines for 
the local institution to adapt to the needs of their local 
community. It provides a detailed topical outline of the 
content of the program, and recommends that each college 
teach this content by giving depth instruction in a major 
procedural language (such as COBOL), some additional in¬ 
struction in a second, or minor, language (such as RPG), 
and a foundation of concepts based on whatever assembler 
language it has available to use. It is recommended that the 
program include an “applied area” of study to be the major 
applications area. This feature of the curriculum recommen¬ 
dations allows the purpose of the program to be toward the 
commercial or business field, toward the scientific area, or 
toward any applied field of interest to the institution or to 
the student. 

The report recommends laboratory experiences of a variety 
of types, with one type in one course, and perhaps another 
type in other courses. It is recommended that immediate 
turnaround of computer runs be provided during the sched¬ 
uled laboratory sessions with the instructor present, with 
several additional turnarounds provided for open laboratory 
use. The report suggests an increased use of timesharing 
access, with experience in interactive programming pro¬ 
vided. It recommends increased emphasis on concepts and 
usage of an operating system during coursework, increased 
emphasis on data and file structures, and increased use of 
pre-programmed packages and utilities. 

The report offers a real challenge to those two-year insti¬ 
tutions interested in maintaining an up-to-date, viable pro¬ 
gram. It will perhaps serve as an incentive to review and re¬ 
examine equipment, facilities, and programs. 

EMPLOYMENT NEEDS 

Students who complete the two-year Associate in Arts or 
Associate in Sciences degree program are generally capable 
of working as a junior computer programmer or program¬ 
mer-trainee. Most graduates go into commercial applications 
work, into banks, insurance companies, small business, etc. 
Others go into statistical data centers, service bureaus, or 
local government data processing. Although most commu- 
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nity and junior colleges are required to perform student 
follow-up studies, most is done on a college-wide basis 
rather than for any particular curriculum. Therefore little 
has been documented on graduates of two year programs in 
computer education. 

Most two-year educators feel that placement of graduates 
has been good. Many graduates have, within three years, 
moved up the promotion ladder into a variety of jobs, rang¬ 
ing from senior programmer, systems programmer, to man¬ 
agement in a small company. Often, within five years of 
graduation, almost a third have not only been promoted on 
the job, but have acquired a baccalaureate degree, generally 
by continuing to an upper division institution on a part-time 
basis. Employment practices are, however, often criticized 
by these community and junior college program coordina¬ 
tors, to the extent that many local businesses and industries 
require that the entry level position be that of an operator, 
sometimes even a peripheral equipment operator. This mis¬ 
match of training to the job is usually “corrected” once the 
applicant has proven valuable. 

More and more baccalaureate programs are becoming 
available in computer education, with the number of them 
recently passing the number of associate level programs. 13 
Many companies now show a definite preference for the 
four year graduate, as does the U.S. Government. Two year 
college graduates generally have their best opportunity, in 
their first job, with the small systems shop. Recently, several 
colleges have specifically attempted to outline a separate 
option within their program to fill that need. 

OPPORTUNITIES AND CHALLENGES 

Students interested in the computer field as a profession 
have opportunities at the community and junior college level 
to learn about the field, gain good technical skills in com¬ 
puter programming, and get a good general education. This 
initial investment is usually enough to give a talented person 
enough foundation to allow continuation in further learning 
while working in the field. Community colleges generally 
cost less, do not require students to move away from home 
while going to school, and in most cases provide a somewhat 
more personalized atmosphere than do the large state col¬ 
leges and universities. 

However, certain problems do exist. Many center around 
the problems arising from the lack of well-qualified faculty. 
It is difficult to find, and keep, faculty with good academic 
credentials, several years of experience doing computer 
work, and several years of teaching experience. With the 
declining enrollments forecast by professional educators, 
administrators are hesitant to hire new full-time faculty, and 
are, instead, promoting “cross-training” of those faculty in 
other departments whose enrollments have already declined. 
This can be done, and can be done well, but it requires more 
time than is generally given. Secondly, problems arise, and 
are increasing, from the lack of proper equipment, or lack 
of proper access to the equipment to be used, or lack of 
control over the types of activities to be done on the equip¬ 
ment. Thirdly, problems exist in the lack of commitment on 


the part of the institution to ensure a quality program. It is 
difficult for instructors teaching full-time to properly revise 
and maintain both a facility and a curriculum, without 
needed released time from course instruction. Fourthly, 
problems exist in the lack of understanding of the students 
about what the career field is, and the types of skills and 
talents that success in the field requires. The mystique of 
the “computer” has not been disspelled from entering stu¬ 
dents. Lastly, problems exist in the transition of the student 
from one level of schooling to another, commonly called the 
“articulation” problem. It occurs between secondary 
schools and community colleges as much as it does between 
community colleges and upper division institutions. 


NEEDS FOR THE FUTURE 

Developments in the computer field continue to occur at 
extremely fast rates. The boom in minicomputer and micro¬ 
processor technology is having a profound effect on business 
and industry. The growth in small diskette and cassette 
computers has led to an increased need for personnel who 
can serve as a combination of operator, analyst, data entry 
operator, and programmer. The effect of data base manage¬ 
ment inquiry systems and the creation of the position of data 
base administrator will change the way the conventional 
computer programmer performs work. 

Educational institutions often react slowly to the need for 
change. Curricula established a decade ago are, in many 
instances, in need of attention in both the content area and 
in the physical facilities being used. Several rather urgent 
needs, given here in prioritized order, exist: 

1. A return to the philosophy of attempting to do those 
things that are “academically” better, rather than 
doing merely those things that are the absolute mini¬ 
mum for retaining the program. We should be looking 
for the best teachers, the best in facilities and the best 
access to them, and strengthening of the content of the 
programs, both technically and otherwise. Getting this 
is NOT always a matter of increasing costs. 

2. An emphasis on a well-qualified, kept-up-to-date fac¬ 
ulty, given time for curriculum development as well as 
for teaching. There should be opportunities for “cross- 
trained” teachers to gain experience by working under 
the auspices of the college, perhaps, before being 
placed technically “over their heads” into a classroom. 
Special programs should be considered for community 
college instructors, perhaps on a state-wide or regional 
basis. Opportunity to interact with other teachers, as 
is possible in the ACM’s Special Interest Group for 
Computer Science Education, 14 should be found. Ac¬ 
cess to recent computer literature should be planned, 
by means of journals, newsletters, etc. 

3. More imaginative use of the monies spent on computer 
equipment. Recently decreased costs of equipment can 
make an older five-year lease very expensive. Alter¬ 
native types of computers should be studied and con- 
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sidered for use in an entire curriculum, or in one 
course. Access to this equipment should be planned 
under the philosophy recommended in 1. above. 

4. An increased emphasis on career education in the com¬ 
puter field, from the middle school level, at least. Bet¬ 
ter advising and counseling is needed in both secondary 
schools and community colleges. Encourage profes¬ 
sional societies to revise career leaflets as they become 
dated, and urge students and counselors alike to use 
career books presently available. 15,16 

5. Availability of a “certification” examination suitable 
to the requirements of the entry-level computer pro¬ 
grammer. Such an examination would give potential 
employers a reference point for the evaluation of the 
knowledge base of the entry-level applicant. It would 
provide a means for continuous professional over-view 
of what the entry-level person should know, as contin¬ 
uing re-evaluation of the examination occurs from year 
to year. It would provide the community and junior 
colleges with some statistical data on the scores of 
their graduates for continuous evaluation of their pro¬ 
gram. 

OUR ALTERNATIVES 

Community colleges have experienced explosive growth 
over the last few decades. Programs established during that 
period are in need of attention. Those that get it, and a 
rebirth of commitment, should survive, and will perform a 
service to society in the preparation of well-qualified persons 
for the use of computers in the future. Those programs that 
continue without changing to meet industry requirements 
will, in due time, slowly fade away. The needs will be met 
somewhere else, perhaps by a program at the four-year 
level, in the state colleges or universities. The decision, and 
the challenge, is ours. 
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INTRODUCTION 

The use of computers in business is increasing dramatically, 
both in the diversity of applications and in the number of 
smaller firms now employing them. Hence, one would ex¬ 
pect a corresponding increase in the number and extent of 
combined business/computer science programs in under¬ 
graduate college and university curricula. In an effort to 
explore this increase, and to provide one resource for their 
work on a combined program at California State University, 
Chico, the authors did a survey of such programs in the 
United States. This paper is a report on the results of that 
survey. 

METHODOLOGY 

The survey was conducted during the 1976-77 academic 
year. Program directories, departmental brochures, cata¬ 
logs, professional journals, mail inquiries, and personal dis¬ 
cussions were used to compile an initial “candidate” list of 
schools and their programs. 

Sub-minimal crossovers were eliminated during this com¬ 
pilation; “minimal” was arbitrarily taken to mean “a com¬ 
bined program in which the degree area required at least 75 
percent of the units normally required for a minor in the 
other area;” e.g., if a typical minor was 20 units, then a 
business program which required fewer than 15 units of 
computer-related courses was considered sub-minimal. 

As indicated above, this cutoff was arbitrary. It was un¬ 
fortunately necessitated after initial work indicated a very 
large candidate list, but one in which a significant number 
of supposedly combined programs required only two or 
three courses in the non-degree area. 

It is interesting to note that during the search process the 
perspective of the survey changed. It was originally intended 
to have two aspects: computer science majors with business 
options and business majors with computer science options. 
However, the authors were able to identify very few of the 
former; hence, the focus is on the latter. 

After compilation of the candidate list, work was begun 
on an in-depth analysis of each program and the courses in 
it. At this stage some programs were unfortunately dropped 
from consideration because precise requirements and/or de¬ 


scriptions for course numbers were not available in time. 
The final sample consisted of 56 schools offering a business 
degree with a minimal option in computer science. 

Results were tabulated according to a course list derived 
from the union of all the samples’ courses. Since emphasis 
here is mainly on content rather than quantified amounts, it 
was decided not to distinguish between quarters, semesters, 
4-1-4, etc. Similarly, only two status categories were used: 
required and elective. Required courses are those explicitly 
and individually specified. Elective courses comprise all oth¬ 
ers, including such listings as “select two of the following 
six,” etc. 

The percentage of schools having each course required, 
elective, and either required or elective is shown in Appen¬ 
dix I. Appendix II is a selection of those courses most often 
required. Finally, Appendix III shows several of the more 
comprehensive programs. 

COMMENTS 

It must be stressed that the fundamental orientation of 
this survey has been the general composition of computer 
science options in business curricula. Ideally, of course, an 
advisor from every school would have been interviewed 
with regard to the program in his/her department. Limited 
resources, time constraints, and the number of programs 
involved made this impossible. Therefore, although every 
possible effort was expended to insure accuracy and cpm- 
pleteness, no claims can be made as to the inclusion of all 
programs or the exact specification of programs included. 

The survey results have proven very useful for the au¬ 
thors’ purposes. It should be of similar use for others de¬ 
signing combined programs. 

In closing, two sidelights were of interest to the authors. 
First, the large number of courses coupled with the small 
percentage of schools requiring many of these courses seems 
to indicate considerable diversity of opinion as to which 
topics should be part of such a program. Second, the topic 
“security” was automatically entered at the start of the 
tabulations. Given the concern with computer crimes and 
abuses, it was surprising to find that none of the programs 
included a course in security as either a requirement or 
elective. 


1209 



1210 


National Computer Conference, 1978 


APPENDIX I—REQUIRED AND ELECTIVE PERCENTAGES 


I. Introduction and Languages 

REQUIRED 

ELECTIVE 

TOTAL 

1. Introduction to computers 

85.7 

1.8 

87.5 

2. Flowcharting 

3.6 


3.6 

3. Documentation 

1.8 


1.8 

4. Basic 

1.8 


1.8 

5. FORTRAN I 

37.5 

3.6 

41.1 

6. FORTRAN II 

5.4 

3.6 

8.9 

7. R.P.G. 

8.9 


8.9 

8. COBOL I 

51.8 

7.1 

58.9 

9. COBOL II 

10.7 

5.4 

16.1 

10. PL/I 

7.1 

3.6 

10.7 

11. Assembly I 

30.4 

8.9 

39.3 

12. Assembly II 

1.8 

1.8 

3.6 

13. Survey of Languages 

8.9 

7.1 

16.1 

14. Programming Applications 

23.2 

5.4 

28.6 

15. Advanced Programming 

5.4 

1.8 

7.1 

II. Foundations 




1. Data Structures 

19.6 

5.4 

25.0 

2. Numerical Analysis 

1.8 

1.8 

3.6 

3. Discrete Structures 

5.4 


5.4 

4. Information Theory 

1.8 


1.8 

5. Automata Theory 

1.8 


1.8 

6. Symbolic Logic 

1.8 


1.8 

7. Analysis of Algorithms 

7.1 


7.1 

8. Formal Languages 

3.6 


3.6 

9. Systems Theory 

1.8 

1.8 

3.6 

III. Hardware and Operations 




1. Computer Organization 

5.4 

3.6 

8.9 

2. Hardware Systems 

3.6 

3.6 

7.1 

3. I/O Devices 

1.8 


1.8 

4. Systems Programming 

12.5 

1.8 

14.3 

5. Compilers 

1.8 


1.8 

6. Operating Systems I 

8.9 

3.6 

12.5 

7. Operating Systems II 


3.6 

3.6 

8. Real Time Systems 

5.4 

3.6 

7.9 

9. Selection of Hardware Systems 

3.6 


3.6 

10. Use of Software Packages 

1.8 


1.8 

11. Analogs 

1.8 


1.8 

12. Minicomputers 

1.8 


1.8 

13. Management of Data Processing Systems 

8.9 

1.8 

10.7 

14. Person/Machine Interaction 


1.8 

1.8 

15. Security 




IV. Information Systems 




1. Information Systems I 

17.9 

1.8 

19.6 

2. Information Systems II 

3.6 


3.6 

3. Accounting Systems 

5.4 

1.8 

7.1 

4. Management Information Systems 

21.4 

3.6 

25.0 

5. Systems Analysis 

46.4 

5.4 

51.8 

6. Systems Design 

41.1 

1.8 

42.9 

7. Files and Data Management 

16.1 

3.6 

19.6 

8. Information Retrieval 

7.1 

1.8 

8.9 

9. Database Management Systems 

8.9 

1.8 

10.7 

10. Case Studies 


1.8 

1.8 

V. Probability and Statistics 




1. Probability 

14.3 

1.8 

16.1 

2. Statistics I 

28.6 

3.6 

32.1 
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VI. 


VII. 


VIII. 


3. Statistics II 

4. Sampling 

5. Regression 

6. Non-Parametric Statistics 

7. Multi-Variate Analysis 

8. Statistical Experiments 

9. Reliability Statistics 

10. Queing Theory 

11. Decision Theory 

12. Forecasting 
Operations Research 

1. Operations Research I 

2. Operations Research II 

3. Optimization 

4. Linear Programming 

5. Non-Linear Programming 

6. Dynamic Programming 

7. Integer Programming 
Simulation and Modeling 

1. Simulation 

2. Simulation Languages 

3. Modeling 

4. Deterministic Modeling 

5. Stochastic Modeling 

6. Continuous Systems 

7. Discrete Systems 
Computer Science Topics 

1. Symbol Manipulation 

2. Natural Language Processing 

3. Computer Assisted Instruction 

4. Artificial Intelligence 

5. Pattern Recognition 


14.3 

3.6 

17.9 

1.8 

1.8 

3.6 

3.6 


3.6 

3.6 

3.6 

7.1 

1.8 

1.8 

3.6 


1.8 

1.8 

1.8 

1.8 

3.6 


1.8 

1.8 

10.7 


10.7 

1.8 

3.6 

5.4 

23.2 

5.4 

28.6 

5.4 

8.9 

14.3 


1.8 

1.8 


1.8 

1.8 


1.8 

1.8 


1.8 

1.8 


1.8 

1.8 

21.4 

19.6 

41.1 


1.8 

1.8 

5.4 

1.8 

7.1 

3.6 


3.6 

3.6 

1.8 

5.4 


1.8 

1.8 

1.8 

1.8 

3.6 


1.8 

1.8 

1.8 


1.8 

1.8 


1.8 

3.6 

1.8 

5.4 

1.8 


1.8 


APPENDIX II—TOP 18 REQUIRED COURSES 


Name % Requiring 

Introduction to Computers 85.7 

COBOL I 51.8 

Systems Analysis 46.4 

Systems Design 41.1 

FORTRAN I 37.5 

Assembly I 30.4 

Statistics I 28.6 

Programming Applications 23.2 

Management Information Systems 21.4 

Simulation 21.4 

Data Structures 19.6 

Information Systems I 17.9 

Files and Data Management 16.1 

Probability 14.3 

Statistics II 14.3 

Systems Programming 12.5 

COBOL II 10.7 

Decision Theory 10.7 


APPENDIX III—SOME SAMPLE PROGRAMS 
I. Required: 

Introduction to Computers 

FORTRAN I and II 

COBOL I and II 

Assembly I 

Data Structures 

Analysis of Algorithms 

Management Information Systems 

Systems Analysis 

Systems Design 

Files and Data Management 

II. Required: 

Introduction to Computers 
FORTRAN I 
COBOL I 
Assembly I 
Data Structures 

Management Information Systems 
Statistics I and II 
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Sampling 
Regression 
Decision Theory 
Operations Research I 

III. Required: 

Introduction to Computers 
Survey of Languages 
Programming Applications 
Data Structures 
Discrete Structures 
Symbolic Logic 
Computer Organization 
Compilers 
Probability 
Statistics I and II 
Forecasting 

IV. Required: 

Introduction to Computers 
COBOL I 
Assembly I 
Data Structures 
Discrete Structures 
Information Theory 
Automata Theory 
Systems Programming 
Information Retrieval 


Natural Language Processing 
Computer Assisted Instruction 
Artificial Intelligence 
Pattern Recognition 

V. Required: 

Introduction to Computers 
FORTRAN I 
COBOL I 

Information Systems I and II 
Files and Data Management 
Statistics I 
Simulation 

Deterministic Models 
Stochastic Models 

VI. Required: 

Introduction to Computers 
Data Structures 
Analysis of Algorithms 
Computer Organization 
Minicomputers 
Systems Analysis 
Systems Design 
Statistics I and II 
Simulation 

Deterministic Models 
Stochastic Models 




A brief survey of computer 
science and engineering education 


by C. V. RAMAMOORTHY 

University of California 

Berkeley, California 

INTRODUCTION 

In this paper we shall attempt an overview of some signifi¬ 
cant developments that have taken place over a span of 
three decades in the Computer Science and Engineering 
(CSE) education and to discuss some very critical issues 
that face the educators and the professional societies. Unlike 
other technologies, the computer technology has been in a 
state of explosive evolution. It has never been easy to define 
or even agree upon what should constitute an essential core 
of knowledge a student should obtain (learn) during a four- 
year period of undergraduate study of the computer tech¬ 
nology. 

With the development of the first electrical computers, 22 
Harvard (1937), Bell Telephone Laboratories (1937) and the 
University of Pennsylvania (1946), the computer industry 
has grown by leaps and bounds. Computer sales currently 
amounted to more than 10 billion dollars in 1975 and it is 
predicted that the industry will experience a 19 percent 
growth in revenues by 1985, with more than two million 
people employed in computer related jobs. One cannot for¬ 
get that we as a nation are wholly dependent on the com¬ 
puter. It becomes therefore, very important for us to study 
the role and trends of CSE education especially as it at¬ 
tempts to satisfy the growing need for well-trained profes¬ 
sionals in this area. 

HISTORICAL DEVELOPMENTS 

It is not easy to trace the evolution of CSE education. 
Universities such as Harvard, MIT, Pennsylvania, and Illi¬ 
nois were probably among the first to include CSE courses 
in their regular curricula. Two types of courses were most 
prominent; the logic design courses primarily emphasizing 
the computer design techniques based on the switching the¬ 
ory and circuit design, and programming courses emphasiz¬ 
ing machine and assembly language programming. While the 
former were generally taught by electrical engineering pro¬ 
fessors, the programming courses were taught usually by 
the specialists of the university’s computing center under 
the auspices of the mathematics department. As the need 
for computer engineers and computer programmers in¬ 


creased, more and more courses were added and a distinc¬ 
tive pattern emerged in the administration of these courses. 
Electrical engineering (EE) departments naturally included 
computer design courses in their curriculum whereas the 
mathematics departments generally handled programming 
and numerical analysis courses. 

The early computers such as Harvard Mark I, ENIAC 
etc., were built through the support of the U.S. Army and 
were intended to calculate the ballistic trajectory tables. 
This heralded the increasing use of computers in scientific 
computations, which in turn placed demands for more col¬ 
lege graduates with numerical analysis and computer pro¬ 
gramming backgrounds. Consequently, mathematics depart¬ 
ments initiated scientific programming courses in their 
curricula, and expanded their numerical analysis programs. 
Computer science (CS) departments began to appear in the 
middle sixties generally as off-shoots of mathematics de¬ 
partments in the Colleges of Arts and Science. Stanford 
University’s Department of Computer Science under the 
late Prof. Forsythe was among the earliest. At present, the 
CSE education in major universities and colleges are admin¬ 
istered mainly either by separate Computer Science and 
Electrical Engineering Departments, or by a combined Elec¬ 
trical Engineering and Computer Science Department. The 
former pattern can be seen in universities like Illinois at 
Urbana, Stanford, and Texas at Austin. The latter plan of 
a combined department is credited to be originated at Uni¬ 
versity of California at Berkeley and at MIT, 19 and is fol¬ 
lowed at universities like Columbia, Northwestern, etc. The 
hardware design oriented courses are taught very naturally 
by the EE departments whereas the CS departments em¬ 
phasize theory and programming. There are, of course, ex¬ 
ceptions to all the above trends. Recently a few EE depart¬ 
ments identified computer engineering as one of their major 
components and call themselves Electrical and Computer 
Engineering Departments, e.g.. University of Michigan, Ann 
Arbor and University of Wisconsin, Madison. Also, CS pro¬ 
grams are sometimes included in Departments of Mathe¬ 
matics, or Statistics or Industrial Engineering. 

We have used the terms computer science (CS) and com¬ 
puter engineering (CE) without defining them. Perhaps the 
distinction between them can best be made by taking a 
closer look at each. The computer scientist is interested in 
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the theory and science of computation and programming. 
Thus, areas such as automata theory, formal theory of lan¬ 
guages, complexity theory, numerical analysis and the math¬ 
ematical foundations of algorithms, and data structures 
(thus, the science of programming a la Wirth) cover the 
discipline of computer science. The computer engineer is 
interested in the specification, design, implementation, and 
utilization (operation) of data processing systems including 
both hardware and software. Thus, computer engineering 
can be defined as that branch of engineering concerned with 
the organization, design, and utilization of digital processing 
systems as general purpose or specialized computers or as 
components of larger systems. 2 Thus the computer engineer 
(including the software engineer) uses the principles of com¬ 
puter science and/or electrical engineering in specifying, de¬ 
signing, implementing, and utilizing computer systems for 
specific applications. 

CURRICULUM DEVELOPMENT 

Three basic milestones can be identified in the curriculum 
development of the CSE education. They are (1) ACM Cur¬ 
riculum 68, (2) COSINE Committee recommendations, and 
(3) Computer Society’s Model Curriculum Subcommittee 
(CSMCS) recommendations. 

(1) ACM Curriculum 68 —The ACM’s Curriculum Com¬ 
mittee on Computer Science (C 3 S) published a set of rec¬ 
ommendations called the Curriculum 68, 1 in March 1968. 
This was a comprehensive set of proposals for four-year 
undergraduate programs and some graduate programs. It 
also included description of contents of supporting courses 
in mathematics. In a previous set of recommendations, the 
committee devoted considerable attention to the justification 
and description of computer science as a discipline. Curric¬ 
ulum 68 was adapted by several universities as a model for 
their computer science programs. Its major effort had been 
to help standardize the undergraduate and graduate pro¬ 
grams in many CS departments. The Curriculum 68 provided 
a clear description of course contents together with pertinent 
technical references. It also specified a sequence of core or 
essential courses that should be taught in a four-year un¬ 
dergraduate curriculum. Aiming primarily at programming 
and mathematically oriented computer science areas, it 
shied away from any recommendations for engineering ori¬ 
ented computer education. 

(2) COSINE Committee —In the middle sixties, the COS¬ 
INE Committee of the National Academy of Engineering 
studied the computer options in Electrical Engineering De¬ 
partments. In a series of reports, they discussed various 
aspects of computer education in electrical engineering, in¬ 
cluding courses in computer organization, 4 digital subsys¬ 
tems, 3 * 5 digital systems laboratories, 6 and minicomputers. 8 
An important COSINE report known as the Coates Com¬ 
mittee’s report 2 defined the computer option in electrical 
engineering which set forth a basis for the development of 
many future computer engineering programs. Also, the 
COSINE Committee’s report on operating systems 7 pro¬ 


vided the first comprehensive course description in that 
important area. Judging from the number of electrical engi¬ 
neering departments having computer science and engineer¬ 
ing options patterned after COSINE Committee reports, it 
can be said that COSINE has been quite successful. The 
COSINE Committee’s workshops and reports helped to in¬ 
itiate several faculty in EE department into computer engi¬ 
neering and helped make CE a major component of EE. 

(3) Computer Society’s Efforts —The Model Curricula 
Subcommittee of the Computer Society’s Education Com- . 
mittee started efforts in 1974 to develop a four-year curric¬ 
ulum in CSE. “Development of model curricula that mesh 
computer science and engineering has, over the past decade, 
been a tar pit and many great and powerful beasts have 
thrashed violently in it.” 10 The Subcommittee divided the 
CSE program into four subject areas and have provided 
course definitions and contents in each. The subject areas 
are digital logic, computer organization and architecture, 
operating systems and software engineering, and theory of 
computing. It is evident that these subject areas are quite 
broad and no single four-year curriculum could treat them 
adequately. Hence, the Subcommittee has proposed model 
curricula which are formed with elements considered essen¬ 
tial in each of these subject areas. A number of options in 
the proposed curricula would provide some specialization. 
Increased specialization is provided at the graduate level. 

In a way, the Computer Society’s Model Curriculum ob¬ 
jectives were different from those of ACM’s Curriculum 68, 
the major difference being that the former recognized the 
need for both computer science and computer engineering 
in a single curriculum. In that sense, the Computer Society’s 
Model Curriculum recommendations “define curricula that 
are interdisciplinary in nature with emphasis placed both on 
computer engineering and computer science.” This was also 
the first effort to bridge the gap between computer science 
and computer engineering and hardware and software. To 
date, a number of reports have been published by the Model 
Curricula Subcommittee which discuss some critical issues 
and rationale for their recommendations. 10,11 Another sig¬ 
nificant contribution of the Computer Society is the report 
on Computer Architecture Curriculum. 12 In this study sev¬ 
eral well-known educators, engineers, and scientists have 
identified the functions and the role of the computer archi¬ 
tect and then proposed a broad based curriculum consisting 
of both computer science and computer engineering subjects 
to educate future computer architects. 

In effect, the recommended curricula stress good basic 
scientific and engineering education which provide the stu¬ 
dent the potential to continue education all through the 
professional career to assimilate and use the advances of 
computer technology. 

DISE committee’s work 

Digital Systems Education Committee (DISE) was formed 
in 1974 with the support of National Science Foundation 
with the object of providing quality educational materials in 
the general area of digital systems. The purposes of the 
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project and the goals are [CAIN]: (1) to develop and/or 
coordinate the development of educational/instructional ma¬ 
terials in the digital systems area; (2) to achieve widespread 
dissemination and use of developed materials; and (3) to 
encourage, develop and facilitate mutual cooperation and 
information transfer between the academic and industrial 
sectors. The ultimate objective of the DISE project was to 
help the educators to keep the courses and curricula current 
and in step with the rapid technological and theoretical de¬ 
velopments. The committee solicited and supported propos¬ 
als for developing educational materials such as textbooks, 
lecture material, videotapes, and self-paced instruction 
books in various digital system topics. The Committee also 
provided support for review, evaluation, and testing of these 
materials. A repository was established under the direction 
of Prof. C. L. Coates of Purdue University to collect un¬ 
published digital systems educational material and to dis¬ 
seminate information regarding the items in the collection. 
DISE Committee has closely collaborated with the Com¬ 
puter Society’s Model Curriculum Subcommittee using the 
latter’s curriculum recommendations for developing their 
educational materials. They sponsored workshops on Mi¬ 
croprocessors and Education which brought the industry 
and university engineers together to exchange latest tech¬ 
nological information. 


CONCLUSION 

An extended and more comprehensive evaluation of CSE 
education will appear in the September issue of the IEEE 
Proceedings. 
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A panel session—Accreditation of computer-oriented 
academic programs 


SESSION CHAIRMAN—GERALD E. WAGNER 

California State Polytechnic University, Pomona 


Panel Members 

Thomas H. Athey—California State Polytechnic 
University 

Sonya L. (Sam) Anderson—T.H. Gamer Company 

Donald B. Medley—Moorpark College 

Eugene B. Smith—U.S. Department of Agriculture 

PANEL OVERVIEW—Gerald E. Wagner 

The term “accreditation” is defined by the Standard College 
Dictionary as, “The granting of approved status to an aca¬ 
demic institution by an accrediting body after an examina¬ 
tion of its courses, standards, etc.” Traditionally different 
levels of accreditation have been available. For example, 
individual degree programs (majors) can be accredited, total 
programs can be accredited such as is provided by the 
American Assembly of Collegiate Schools of Business 
(AACSB), and entire university programs can be accredited. 

This panel will limit itself to the topic of accreditation as 
it applies to a single academic program. The actual name of 
the computer degree program is insignificant for the pur¬ 
poses of this presentation and all computer-oriented degree 
programs are included whether they be called “Computer 
Science,” “Business Data Processing,” “Information Sys¬ 
tems,” “Information Science,” or something else. The en¬ 
tire subject or accreditation will be reviewed—both pro and 
con—as it relates to these programs. 

Any decision to accredit or not to accredit academic com¬ 
puter degree programs could have a serious impact on many 
different groups within our society. The positive and nega¬ 
tive aspects of accreditation will be discussed as it relates 
to: (1) students (both Community College and University), 
(2) degree-granting institutions, and (3) future employers of 
the graduates of the degree programs. 

Many different professional organizations have attempted 
to define curricular patterns, but as of this date, no attempt 
has been made to provide a method for evaluating academic 
programs. These past efforts have not been wasted though 
and any efforts to develop accreditation “standards” could 
and should utilize much of knowledge gained through ex¬ 
periences of these other groups. Some of the relevant efforts 
in this area will be discussed as they relate to the accredi¬ 
tation process. 


Further insight may be gained regarding the accreditation 
process by reviewing the history of other accreditation 
groups. In particular, the American Assembly of Collegiate 
Schools of Business and the Engineers Council for Profes¬ 
sional Development will be discussed for that purpose. 

Although accreditation has its inherent problems, the ben¬ 
efits to be gained appear to outweigh any of the potential 
problems or disadvantages that may result. The panel mem¬ 
bers recommend that immediate steps be taken to develop 
a workable accreditation process. The first step to be taken 
would be to establish a committee with representatives from 
business, professional associations and academic institu¬ 
tions. Furthermore, it is recommended that any efforts to 
develop accreditation standards must recognize the different 
roles that different types of computer degree programs play 
in our society. Any standards must be designed to encourage 
creativity—not stifle it—as long as the “creativity” does not 
detract from or change the Overall objectives of the academic 
program. 


ACCREDITATION OF COMMUNITY COLLEGE 
DATA PROCESSING PROGRAMS—Don B. Medley 

Black’s Law Dictionary defines accreditation as “To send 
with credentials as an envoy.” The New Century Dictionary 
by Appleton-Century-Crofts, Inc. defines accredit as fol¬ 
lows: 

To bring into credit; invest with credit or authority; also 
to send with credentials; also, to give credence to; believe; 
trust; also, to credit with something ascribed. 

My concern is that we in higher education are sending our 
graduates out with credentials as an envoy and the creden¬ 
tials are being rejected. Considerable effort has been ex¬ 
pended by the government and the several professional or¬ 
ganizations in the definition of curriculum patterns for 
programs at all levels of higher education but little has been 
done to accredit or validate the programs after they are 
implemented. Probably the most active organization to 
work on curriculum development and design has been the 
Association for Computing Machinery (ACM). A curriculum 
working paper containing recommendations and guidelines 
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for a community and junior college career program in com¬ 
puter programming was published in the Special Interest 
Group on Computer Science Education (SIGCSE) bulletin 
in June, 1977. 

The paper addresses the courses required in such a pro¬ 
gram and the detail content of each course is defined. The 
instructional resources, equipment and faculty needs are 
covered but the only comments related to accreditation deal 
with articulation. While this developmental work and the 
followup effort defined in the articulation section are ex¬ 
tremely important and must be continued and expanded, 
there is also a pressing need for a formal evaluation program 
that will accredit the offerings of particular institutions. 

An evaluation system for community college occupational 
program (COPES) has existed in California since 1971. 
COPES is a system for voluntary evaluation of occupational 
programs administered through the Chancellor’s Office, Cal¬ 
ifornia Community Colleges in Sacramento, California. 
Among the top strengths of the COPES evaluation as rated 
by the COPES teams are: 

• Qualifications of instructional staff 

• Occupational experience of instructors 

• Quality of occupational instruction 

• Administration’s commitment to occupational educa¬ 
tion 

• Adequacy and availability of instructional materials 

• Updating of instructional content and method 

• Utilization of instructional facilities and equipment 

Another case for accreditation standards is developing 
with the expansion of the programs offering college credit 
for “Life Learning and Experience’’. Perhaps the profes¬ 
sional associations could work with the College Entrance 
Examination Board and have the information systems in¬ 
cluded in their long-range examination of American educa¬ 
tion called Future Directions for a Learning Society. 


ACCREDITATION: PROBLEMS AND 

PERSPECTIVE—Eugene B. Smith 

Computer Science education has evolved over the last 25 
years to a significant element of many academic programs. 
The level of training varies from introductory material for 
all students to the research oriented PhD degree. The scope 
of the material covered and the location of the instruction 
within the Univeristy vary widely. The level of maturity of 
this field dealing with computers and their use has reached 
a point where there should be a concerted effort to initiate 
some predefined measure of quality control over the instruc¬ 
tion being provided. Accreditation is an accepted vehicle for 
identifying and maintaining a specified level of quality for 
academic programs. 

If one were to contemplate advocating accreditation for 
computer related educational programs, it is important to 
understand what the term accreditation implies. This paper 


will provide a discussion of the functions of accreditation, 
the types of accreditation, and the accreditation procedure. 

Educational institutions in the United States are not con¬ 
trolled by a single Federal ministry. Accrediting bodies in¬ 
clude governmental agencies, regional associations, and pro¬ 
grammatic associations. Both institutions and program areas 
within institutions may receive various types of accredita¬ 
tion. Some consideration must be given to the levels of the 
institution which would be included in any accreditation 
effort. 

A brief discussion of other accreditation groups should 
provide some insight into the approach which could be taken 
in implementing this type of effort. References to the history 
of the American Assembly of Collegiate Schools of Business 
(AACSB) and The Engineers Council for Professional De¬ 
velopment will be provided. 

A better understanding of the accreditation process and 
identification of the players who must participate in such an 
activity will provide an atmosphere within which potential 
problem areas can be identified. Considering the state and 
scope of computer oriented education, now is the time to 
begin an attempt to develop a workable accreditation proc¬ 
ess. A reasonable first step would be to establish a qualified 
task force with the dedication required to initiate an accred¬ 
itation effort. A coalition of business and academic repre¬ 
sentatives, with the support of the professional associations 
should be formed. 


ACCREDITATION—A UNIVERSITY PERSPECTIVE— 

Thomas H. Athey 

Universities have a major part to play in the training of 
individuals who desire to become EDP professionals. The 
significance of this role will increase in the near future as 
user organizations progress to more sophisticated and en¬ 
compassing computer-based information systems. These 
types of systems will require individuals who have sufficient 
skills at the entry level to make a meaningful contribution 
to the organization plus a broad enough perspective to func¬ 
tion effectively in society. Universities are uniquely quali¬ 
fied to assume this responsibility. 

There are many groups who should be vitally interested 
in the training process performed by the university. These 
stakeholders include employers, associated departments 
within the universities, junior colleges and high schools, and 
students. These stakeholders need reliable evidence con¬ 
cerning the excellence of curriculum and the resultant com¬ 
petency of the graduates who desire to apply these learned 
EDP skills. 

One of the ways of developing indicators of quality and 
competency is through the accreditation process. In the 
past, this has been attempted by several professional orga¬ 
nizations within the EDP field. The initial work has been in 
establishing the resources utilized in the various programs 
to include measurement of the type of equipment available, 
the number of faculty, etc. The next stage in development 
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has been in curriculum building. Establishing the types of 
courses and patterns that seem to be most appropriate for 
acquiring particular EDP skills and perspectives. A most 
ambitious approach has been put forth by the Accreditation 
Research Committee of AACSB. They are concentrating on 
a value added method by which they propose to measure 
competence levels of knowledge, skills, aptitudes, attitudes, 
and personal characteristics of individuals who complete 
various programs. 

While the major advantage of accreditation is the resulting 
verification process, there are several other benefits which 
could accrue from a well designed plan. These would include 

(1) development of more meaningful and effective articula¬ 
tion agreements between junior colleges and universities, 

(2) explicit recognition of the highly different orientations 
required of curriculum aimed at different segments of the 


EDP professional market, and (3) the development of apti¬ 
tude tests which are much more inclusive and valid than the 
present programmer’s aptitude tests. 

The accreditation process is not without its potential dis¬ 
advantages. These include (1) a strong tendency to insure 
uniformity of programs, which could result in stifling inno¬ 
vation, (2) the possibility exists for universities to become 
too vocationally oriented and forget that their mission is 
education, and (3) if any type of mass testing is required, 
not being able to constantly update the testing procedure to 
reflect the needs of the rapidly changing field. 

Overall the case for an accreditation process seems very 
strong. However, to be successfully implemented the accred¬ 
itation body must include representatives from each of the 
major stakeholders in the training process and useful meas¬ 
ures of competency must be determined. 





Area Director: 
Hideo Aiso 
Keio University 
Yokohama, Japan 


Recent progress in Japan 


It is hoped that the area entitled “Recent Progress in Japan” will provide the 
audience with a better understanding of computer technology in Japan. The 
primary objective is to overview the present status and the future prospects of 
both semiconductor and computer technologies from viewpoints of recent activ¬ 
ities in the field of research and development, and to present some main research 
efforts on the development of computers and their application systems suitable 
for Japanese society. 

The first session consisting of two papers is devoted to presenting the state of 
semiconductor technology in Japan. Semiconductor technology is expected to be 
responsible for the next major evolution, not only in the computer industry but 
also in the electronics industry in general. The first paper includes a review of 
the recent advances in electron optical studies relating to electron beam pattern 
generation with sub-micron resolution using the variable shaped beam technique. 
The direct-writing electron beam technology capable of sub-micron geometry is 
considered to be one of the most important steps towards the achievement of 
advanced VLSI ICs for next-generation computers. The second paper presents 
research and development activities in both academic laboratories and semicon¬ 
ductor industries in Japan. It also explores new semiconductor devices offering 
excellent high-speed switching characteristics with low power dissipation, which 
are expected to be used for realizing future computers. 

The second session is intended to overview the development of computers and 
technical progress in their application systems in three separate papers. The first 
paper describes, after a short historical background, the technological character¬ 
istics of newly developed computer series with emphasis on their architectures 
and LSI implementations. The research and development projects of both VLSI 
and PIPS (Pattern Information Processing System), which are organized by com¬ 
puter industries, academic laboratories and the Government are taken up briefly. 
The second paper discusses recent features and future prospects of remote data 
processing in Japan from the viewpoints of various aspects of technology, such 
as development background, computer network architecture or telecommunica¬ 
tion. The final paper discusses the technological trends in real-time application 
systems through describing the design concept of a very reliable operation control 
system for the high-speed railway system called the “Shinkansen.” 
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The third session consists of three papers concerned with recent outstanding 
research efforts on the development and application of relatively small computers. 

The first paper gives a detailed description of a very high-performance NMOS/ 
SOS microprocessor developed as part of the PIPS project. This microprocessor 
contains approximately 7,000 gates in a single chip with 80 pins, and executes 
most microinstructions at the speed of 200 ns. The technology applied to the 
realization of this microprocessor speaks well of the potential power of Japanese 
semiconductor technology today. The second paper describes a very interesting 
application of computers, namely a comprehensive automobile control system 
giving the driver indications of the best routes to take, as well as other information 
such as specific traffic regulations. Moreover, it contributes to economical use 
of gasoline and reduction of air pollution. 

The rapid progress of LSI technology and minicomputers has great potential 
to build very reliable computing systems using distributed processor systems 
concept. The final paper describes how distributed computers can be effectively 
applied to advanced process industries, where control systems, that are more 
reliable than the ones in use now, are needed. This paper discusses the require¬ 
ments, design considerations, advantages and possibilities of these systems from 
the viewpoints of both hardware and software developments. 



Electron beam lithography 
for advanced LSI fabrication 

by Eli CHI GOTO 

University of Tokyo and The Institute of Physical and Chemical Research 
Wako-shi, Saitama, Japan 

and 

TAKASHI SOMA, MASANORIIDESAWA and TATEAKI SASAKI 

The Institute of Physical and Chemical Research 
Wako-shi, Saitama, Japan 


INTRODUCTION 

The current technology used in IC (Integrated Circuit) and 
LSI (Large Scale Integration) fabrication is based on optical 
lithography or the art of photocopying. The light wave length 
sets an ultimate and absolute limit on the resolution of the 
optical method at about one micron (/Am=10 -6 meter) and 
the current LSI technology has almost reached this limit. 

In microscopy, a drastic improvement of resolution was 
achieved in the 1930’s by switching from glass (light) optics 
to electron optics. The similar is taking place in lithography 
for LSI fabrication. X-ray lithography and electron beam 
lithography are the two promising methods for improving 
the resolution by at least one order of magnitude over optical 
lithography. The characteristic features of these three lith¬ 
ographic methods are summarized in Table I. 

In Table I, “Mask in Contact” means that a mask, with 
a master pattern inscribed on it, is placed in close proximity 
to the target (silicon wafer covered with photo-resist) so as 
to selectively expose the target to the incident beam. The 
finite and rather short life of the masks in contact is the 
major disadvantage of this scheme. 

In Table I, “Mask Projection” means that an image form¬ 
ing projection system (actually a lens) is placed between the 
mask and the target so that the mask does not have to be in 
contact with the target. Since there is no material suitable 
for making an X-ray projection lens, this scheme is not 
applicable to X-ray. 

In Table I, “Direct Pattern Generation” means a feature 
in which the incident beam is maneuvered so as to generate 
the pattern directly on the target without using a mask. 
While this feature can be implemented by suitably deflecting 
an electron beam at high speed without any fundamental 
difficulties, this is not the case for light and X-rays. Note 
that the sub-micron pattern generation feature is also needed 
for making the masks themselves. Therefore, “electron 
beam pattern generation” is the key and the very heart of 


advanced LSI technology which requires sub-optical or sub¬ 
micron resolutions. 

Moreover, because of flexibility and many other practical 
reasons, electron beam pattern generation is considered ad¬ 
vantageous for making masks to be used with optical lithog¬ 
raphy in many cases. 

The development of practical electron beam pattern gen¬ 
eration schemes is reviewed in a later section. The recent 
developments in electron optics, i.e., the theoretical foun¬ 
dation of the image-forming processes of electron beams, 
are reviewed and their implications to the practical systems 
are discussed. Fly’s eye optics which may find some use in 
the future is also discussed. 


ELECTRON BEAM PATTERN GENERATION 

SCHEMES 

Figure 1 shows three practical electron beam pattern gen¬ 
eration schemes, and their developments are summarized in 
Table II. 

In Figure 1, “Spot Scanning” means that an electron 
beam, with a small circular cross-section (spot), typically of 
0.1 micron in diameter, is deflected electrically (by applying 
magnetic deflection field or electrostatic deflection field or 
both) and the pattern is generated by scanning the beam 
across the area to be exposed to the beam. In other words, 
the pattern is generated similarly to the pictures on a tele¬ 
vision screen, except for much increased resolution. So far 
as scanning and the resolution are concerned, this^scheme 
is similar to SEM (Scanning Electron Microscope). Actually, 
SEM’s with slight modifications were used in the early 
models of pattern-generating machines. For example, the 
first Japanese pattern-generating machine JBX-2A was de¬ 
veloped concurrently with the first SEM, JSM-1. Using this 
spot scanning machine (JBX-2A), Tarui and his collabora¬ 
tors successfully made MOS transistors purely with electron 
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TABLE I.—A Comparison of Lithographic Methods 



“Mask in 
Contact” 

"Mask 

Projection” 

“Direct 

Pattern 

Generation” 

Ultimate 

Resolution 

Optical 

0 

0 

X 

& 1 fim 

X-Ray 

0 

X 

X 

ssO.l fim 

Electron Beam 

0 

0 

0 

sO.l /i,m 


0—Practicable, x—Impracticable. 


beam pattern generation without using any masks at all in 
1968 [LI]. 

EBES2 and VL-1 (entries 3 and 4, Table II) are also “spot 
scanning” machines. In EBES2 the electron beam is de¬ 
flected electrically only in one direction (deflection width 
0.128—0.256 mm) and the target is scanned mechanically in 
the other direction so as to scan and generate two dimen¬ 
sional patterns. In VL-1 the electron beam is deflected two 
dimensionally within a square scanning area of 5 mm by 5 
mm. 

Referring to Figure 1, “Fixed Shaped Beam” means that 
the cross-section of the electron beam at the target has a 
fixed shape, typically a square of 2.5 /xm by 2.5 /xm. This 
is achieved by projecting the image of a mask with a square 
hole on to the target. The pattern is generated by scanning 
and/or patching the area to be exposed to the beam. EL-1 
is a “fixed shaped beam” machine. 

In Figure 1, “Variable Shaped Beam” means that the 
shape of the cross-section of the beam at the target is vari¬ 
able, typically a rectangle x by y /xm with x and y being 
varied from 0.1 to 25 /xm. This is achieved, for example, by 
using two masks each having a square hole. The image of 
the first mask is projected on the second mask and the image 
of the second mask is further projected on the target. The 
combined action of the two masks are varied by the beam¬ 
shaping deflector placed between the two masks, and the 
shaped beam is directed to a selected position on the target 
by means of beam-positioning deflector placed between the 
second mask and the target. The “variable shaped beam” 
concept was first reported at the May 1977 EIPBT (Electron, 
Ion and Photon Beam Technology) Symposium by four in¬ 
dependent groups (entries 6, 7, 8, and 9, Table II). 

Figure 2 is a photograph of one of the first test patterns 
generated by JBX-6A, the variable shaped beam machine of 
the Japanese group. Entry 10 in Table II shows the result of 
the design and feasibility studies made by the authors’ group 
on the variable shaped beam scheme. 

Before making comparison and evaluation of the various 
pattern generation schemes we shall review the recent ad¬ 
vances in electron optics. 

ADVANCES IN ELECTRON OPTICS 

The studies on the image-forming processes of electron 
beams are called electron optics. Similar to glass (light) 
optics, the deviations in the actual image from the ideal 
image are called geometrical aberrations. 


The term “geometrical” means that the wave nature of 
the beam is being neglected. Since the wave nature or quan¬ 
tum mechanical effects can be neglected in the schemes 
shown in Table II, we shall drop the term “geometrical” 
hereinafter for simplicity. The aberration theory of the pure 
(deflection-less) image-forming process of electron beam is 
well established and we can learn the results from definitive 
text books on the subject, such as those authored by Glaser 
[El] and El-Kareh [E2]. 


FIXED SHAPED BEAN 



VARIABLE SHAPED BEAM 
(PATCHING & SCANNING) 



Figure 1—Electron beam pattern generation schemes 



Electron Beam Lithography 


1225 


TABLE II.—Electron Beam Pattern Generators 

0 

Model 

Beam Shape 

w 

Resolution 

(jim) 

Total Beam 
Curr. (A) 

Electrical 

Scan Width 
or Area 

Year 

Reported 

Developed 

at/by 

Reference 

1 

JSM-1 

0.1 

0.1 

10 pA 

lxl mm 2 

1966 

JEOL 

[-] 

2 

JBX-2A 

0.35 <f> 

0.35 

0.01 fiA 

2x2 mm 2 

1968 

JEOL&ETL 

[LI] 

3 

EBES2 

0.25-0.5 4> 

0.25-0.5 

0.1 /iA 

0.128-0.256 mm 

1975 

BTL 

[L4] 

4 

VS-1 

0.05 

0.05 

1 fj. A 

5x5 mm 2 

1975 

IBM 

[L5] 

5 

EL-1 

2.5 □ 

0.4 

3 fiA 

5x5 mm 2 

1976 

IBM 

[L7] 

6 

JBX-6A 

0.5-25 □ 

0.5 

1 fiA 

2x2 mm 2 

1977 

IPCR&JEOL 

[L9] 

7 

_ 

0.6-2 □ 

0.6 

2 fiA 

4x4 mm 2 

1977 

IBM 

[L10] 

8 

— 

0.5 □ □ 

0.5 

0.1-0.4 fiA 

0.256 mm 

1977 

BTL 

[Lll] 

9 

— 

— 

— 

— 

— 

1977 

Thomson 

[L12] 

10 

— 

0.1-25 □ 

0.1 

>1 nA 

7x7 mm 2 

1978 

IPCR 

I-] 


Remarks: 

1. SEM (Scanning Electron Microscope), Japan Electron Optics Laboratory Co., Ltd. 

2. Spot scanning. Electrotechnical Laboratory, Japan. 

3. Spot scanning, Bell Telephone Laboratories Inc., U.S.A. 

4. Spot scanning. International Business Machines Corporation, U.S.A. 

5. Fixed shaped beam. 

6. The Institute of Physical and Chemical Research, Japan. 6,7,8,9,10: Variable shaped beam. 
9. Thomson CSF, France. 


However, when electrical deflection of the beam enters, 
the theories developed up to the 1960’s are all incomplete 
in that the focusing field and the deflection field are spacially 
separated. A unified treatment of spacially superimposed 
focusing and deflecting fields are needed for the full devel¬ 
opment of the aberration theory and this has been accom¬ 
plished only very recently. 

Electron optics is a rather specialized art full of jargon of 
lengthy mathematical formulas and odd terminologies for 
non-specialists. The authors would try their best to review 
the advance without entering the jargon. Nevertheless, we 
shall need the definitions of, at least, some quantities for the 
sake of clarity. 

Figure 3 shows the definitions of three (optical) parame¬ 
ters <*, (3 and y. We assume that there is a final electron lens 
(the objective lens) which projects the beam on the target. 
The shape of the beam on the target is called the image. The 
distance (specifically called the working distance in pattern¬ 
generating machines) from the objective lens to the target is 
denoted by L. 



The parameter a=F/2=a/L, with a being the aperture 
radius of the lens, is the half aperture angle at the image. 
The brightness of an optical lens (say of a camera) is usually 
measured by using an index F called the F-number. Hence, 
a=F/2 means that a is the index representing the brightness 
of the electron lens. 

The parameter fi=b/L, with b being the size of the image, 
is an index representing the size of the electronic image 
relative to the (working) distance L. 

The parameter y=c/L, with c being the amount (or the 
length) of deflection on the target, is an index representing 
the amount of deflection relative to L. 

The ratio AV/V, with V being the acceleration voltage of 
the electron, represents the relative spread in electron en¬ 
ergy, which may be caused by thermal effects on the cath¬ 
ode, variations in the accelerating voltage itself as well as 
by the Boersch effect to be discussed later. 

The aberration theory has to do with expressing the quan¬ 
tity 8w=8(x+i y), the deviation of the actual beam point 



<*«a./L • P=b/L 


a:HALF APERTURE b:HALF IMAGE 
DIAMETER SIZE 



7 = c/L 

C :HALF DEFLECTION 
DISTANCE 


Figure 2—A test pattern generated by JBX-6A 


Figure 3—Three electron optical parameters: a, f3 and y 
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TABLE III.—Advances in Electron Optics of Deflection Systems 


Field 

p* 

D* 

Content (Aberration=Ab.) 

Year 

Author(s) 

References 

1 

M 

M 

3f,f(a,0,-y)=g(a) 

Ab. Elimination 

1971 

Ohiwa,Goto ,Ono 

[E4] 

2 

M 

M 

min | f(a,0,y)| 

Ab. Reduction 

1973 

Owen,Nixon 

[E5] 

3 

M 

M 

/(«,0,y,AV/V) 

Ab. Formula 

1974 

Munro 

[E6] 

4 

M 

M 

min | f(a, 0,y,AV/V)| 

Ab. Reduction 

1975 

Munro 

[E7] 

5 

M 

M 

min | /(a,0,0,AV/V)| 

Ab. Reduction 

1975 

Pfeiffer 

[E8] 

6 

M 

M 

3/,/(«,j3,y,AV/V)=?(a,^,0,AV/V) 


1977 

Goto,Soma 

[E9] 

7 

M 

M 

min | f(a,/3,y,\V/V)\ 

Ab. Reduction 

1977 

Goto,Soma,Idesawa 

[L9] 

8 

ME 

ME 

f(a,/3,y,AV/V) Ab. Formula 


1977 

Soma 

[Ell] 

9 

M 

E 

min | f(a,/3,y,AV/V)\ 

Ab. Reduction 

197x 

Soma [to be published] 



F*:Focusing Field, D*: Deflection Field. 

M:Magnetic, E:Electro-static, ME:Combined Magnetic and Electro-Static. 


incident on the target from the ideal point, as a function of 
a, /3, y and AV/V, where x+iy is the two dimensional 
cartesian coordinate in the target plane in complex form. 
Namely, we have a function / such that 8 h>= f(a,(3,y, AV/ 
V). 

This function is usually expanded into a Taylor series, 
and each term in the series is given a specific name. For 
example, the terms including the energy spread index (AV/ 
V) are called chromatic aberrations. 

The recent advances in the aberration theory and results 
in the design of electron optical systems with reduced ab¬ 
errations are summarized in Table III. 

First, note that the deflection-less (y=0) pure image-form¬ 
ing case has to do with the special case 8w=f(a,fi, 0,AV/ 
V), and that for “spot scanning” the point image (/3=0) 
function f(a, 0,y,AV/V) would be sufficient but the full 
function /(a,/3,y,AV/V) would be needed for “fixed- and 
variable-shaped” beam schemes. 

Ohiwa, Goto and Ono (# 1 in Table III) introduced a MOL 
(Moving Objective Lens) concept and showed that there 
exists (3 f) a system for point image (/3=0) free of all de¬ 
flection-induced aberrations, i.e., /(a,0,y,0)=g(a). Chro- 


SPHERICAL 

ABERRATION 



AXIAL CHROMATIC 
ABERRATION 



Figure 4—Spherical and axial chromatic aberration 


matic aberrations, however, were not included (assuming 
AV/V=0). 

Munro (#3) derived the formula for point image case 
(/3=0) with chromatic aberrations. 

Goto and Soma (#6) derived the formula for the finite 
image case (jS^O) and also theoretically proved the exis¬ 
tence of systems free of all deflection-induced aberrations 
by making use of the MOL concept. 

Soma (#8) derived the most general formula valid for any 
combination of magnetic and electro-static fields and for 
electrons running at relativistic speeds. The formula is some 
80 pages long and the computerized formula manipulation 
system Reduce 2 [E12] was utilized for the deriviation. It is 
extremely difficult to handle such a long formula without 
such manipulation systems. 

The other entries in Table III with “min” (for minimiza¬ 
tion) gave practical results in reducing various aberrations 
(there are some 20 aberrations.) 

We want to emphasize that, in all the aberration reduction 
calculations made by the authors’ group (entries 1, 6, 7 and 
9 in Table II), we also required the beam to land vertically 
on the target. As a result of these studies, all aberrations 
except for two have been found to be either eliminable or 
negligibly small in practice. The two are the spherical ab¬ 
erration 8w s and the axial chromatic aberration 8w 0 , which 
have the following forms: 

8w s = SaP, 8w a =AaAV/V 

where S and A are the aberration constants. Note that these 
aberrations exist even in the deflection-less point image case 
(j3=y=0), i.e., f(a,0,0,AV/V)=Sa 3 +Aa(AV/V). In spite 
of a great number of efforts trying to eliminate the spherical 
aberration, no practical scheme has been found yet. [E3] 
These two aberrations are explained pictorially in Figure 4. 
The spherical aberration means that electrons having larger 
aperture angle a are bent stronger by the lens, and the axial 
chromatic aberration means that slower electrons (AT/ 
V<0) are bent stronger by the lens. 


LIMITS ON CURRENT DENSITY AND EXPOSURE 
SPEED 

Besides geometrical aberrations there is another factor, 
not having its counterpart in light optics, which causes 









Electron Beam Lithography 


1227 


broadening in the shape of electron beams. That is the elec¬ 
tron-electron interaction. When the intensity of an electron 
beam is increased beyond a certain limit the electrons would 
start to repel each other and collide with others thereby 
broadening the shape of the beam. The theory of such broad¬ 
ening is not in a fully satisfactory form [B2] at present. 

Among others the Boersch effect, named after the first 
observer of the effect, [Bl], is a controversial issue. Some 
experimentalists have even questioned the very existence of 
the effect itself. [B5, B6] Boersch and some others [B6] 
observed broadening in electron energy (AV/V) which is 
much larger than the thermal fluctuations to be expected 
from the cathode temperature, when electrons are passed 
through a spacial region of high electron density. Boersch 
effect would imply an increase of chromatic aberrations. In 
addition to the Boersch effect, the space charge causes the 
displacement of electron trajectories. By regarding the elec¬ 
tron beam as a continuous and charged fluid, the displace¬ 
ment caused by the average (averaged over all electron 
trajectories) space charge effect can be easily derived, yield¬ 
ing the following to be called the space charge broadening 
formula: 

8w sc = KIL/(Vva), (in CGS units), 

where a is the half aperture angle, V is the accelerating 
voltage and L is the path length of the electron as defined 
in the previous section. Further, I is the total beam current 
(n ot the c urrent per unit area), v is the electron speed (v= 
\fleVJm), and K is a number of the order of unity, which 
does not depend critically upon the image size /3 nor on 
other detailed structures of the system. Thus, we may well 
use K= 1 as the rule of thumb. 

The experiments performed on JBX-6A (Table II, #6) 
indicate that axial chromatic aberration and chromatic ab¬ 
erration due to the Boersch effect could hardly affect the 
projected 0.1 micron resolution and that the observed broad¬ 
ening of the image agrees reasonably well with the space 
charge aberration 8w sc . Thus, the spherical aberration 8w s 
and the space charge aberration 8w sc are concluded to be 
the major limitations imposed on the resolution and on the 
beam current. 

The space charge aberration 8w sc = KIL/(Vva ) would be 
reduced by increasing a, the brightness of the objective lens, 
but this would increase the spherical aberration 8w s = Sa 3 . 
Hence, there is an optimum value of a as a result of com¬ 
promise. The lithographic process requires the acceleration 
voltage V to be around 20 K volt. Using the design figures 
of L=S= 10 cm, we arrived at the conclusion that “more 
than 1 micro ampere beam current would be feasible at 0.1 
micro meter resolution” as indicated in Table II, #10. 

The following are the implications of 1 pA beam current. 

1. The lithographic sensitivity of the advanced electron 
resist material is better than 10~ 6 coulomb/cm 2 . Hence, 
exposure speed of 1 cm 2 /sec would be achieved (20 sec 
exposure time for 2 inch wafer). 

2. If “spot scanning” with 1 pA beam current in a 0.1 
pm spot were used, scanning speed of 10 10 spots/sec 
(10 Giga-Hz) would be needed, which would be ex- 


DOUBLE DEFLECTION FLY'S EYE OPTICS POST DEFLECTION FLY'S EYE OPTICS 



tremely difficult to engineer. For this reason alone, one 
would have to use “shaped beam” schemes in order 
to increase the scanning speed. 

3. Moreover, 1 pA in a 0.1 pm spot implies 10 4 A/cm 1 2 in 
current density. A very bright cathode (e.g., field emis¬ 
sion type) would be needed to realize such high current 
densities, but this would reduce the engineering free¬ 
dom in the cathode design. The current density would 
be greatly reduced in “shaped beam” schemes. 

4. From these considerations just given, “shaped beam” 
schemes are believed to be quite advantageous and 
even a necessity for attaining high exposure speed. On 
the other hand, “spot scanning” may be advantageous 
for lower speed devices because of its structural sim¬ 
plicity. 

FLY’S EYE OPTICS FOR PURE ELECTRICAL 

LARGE AREA SCANNING 

In all the electron beam pattern generation schemes de¬ 
scribed earlier (cf. Table II), the electrically scanned areas 
(typically 5x5 mm 2 ) are too small to cover the entire target 
area of silicon wafers, typically two to five inches in diam¬ 
eter. Hence, the target is placed on a mechanically scannable 
stage so as to back up the electrical scanning. The widening 
of the electrically scanned area is difficult because it would 
imply an increase in the working distance L resulting in an 
adverse effect on the space charge aberration 8w sc = KIL/ 
(Vva). 

Fly’s eye optics may provide a method for scanning a 
large area purely electrically. Figure 5 shows two fly’s eye 
schemes. The “post deflection” type was invented by New¬ 
berry [FI] and the “double deflection” type, by one of the 
present authors. [EG] [F2, F3, F6]. In Figure 5, FEL (Fly’s 
Eye Lens) is illustrated by a matrix of lenses. In both types, 
the electron beam is selectively passed through a specific 
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lens in the FEL by the action of the main deflector MD. The 
beam is further deflected by sub-deflector(s) SD(s), so as to 
scan a small area. The difference between the two schemes 
consists in that while a number of small sub-deflectors are 
placed after the FEL in the “post deflection” scheme, a 
single sub-deflector placed before the FEL is sufficient in 
the “double deflection” scheme. Lenses LI and L2 are 
inserted in the “double deflection” scheme for the explan¬ 
atory purpose but they may be removed. 

The “post deflection” scheme and its modifications are 
used in electron beam memory schemes, with sub-micron 
memory cells. [F4, F5, F7]. The “double deflection” 
scheme is used in very high precision cathode ray tubes. 
[F2, F3, F6], 

CONCLUSION 

Electron beam pattern generators with 0.1 micron resolution 
running at a speed greater than 1 cm 2 /sec are believed to be 
technologically feasible, based on electron optical studies. 
The “variable shaped beam” scheme will be used in faster 
machines and the “spot scanning” scheme, in slower ma¬ 
chines. 

The impacts of such pattern generators on LSI fabrication 
are rather difficult to foresee precisely because of the con¬ 
glomerate nature of the LSI technology. We, as specialists 
in electron optics, firmly believe that the electron beam can 
never be the bottleneck in the development of advanced 
LSI’s with sub-micron and sub-optical patterns. 

We would like to acknowledge Messrs. S. Miyauchi, K. 
Tanaka and T. Someya of JEOL for stimulating discussions 
and for providing us with the photograph of the test pattern 
(Figure 2). 
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INTRODUCTION 

The present status of semiconductor technology in Japan 
will be reviewed with special emphasis on research and 
development of semiconductor devices and of large scale 
integrated circuits for computers. First of all, major pro¬ 
grams of R&D of semiconductor devices and large scale 
integrated circuits will be briefly described and then impor¬ 
tant achievement, especially on semiconductor devices for 
high speed digital application and on large scale integrated 
circuits for logic and memory use, will be discussed. 

MAJOR RESEARCH PROGRAMS 

Research and development of semiconductor devices and 
large scale integrated circuits are active in industry, govern¬ 
mental and semi-governmental laboratories and universities. 

The Electrical Communication Laboratories, Nippon Tel¬ 
egraph and Telephone Public Corporation (NT&T) and the 
Electrotechnical Laboratory, Agency of Industrial Science 
and Technology, are significantly contributing to R&D of - 
LSI through their own work and also through their projects, 
although their activity is not limited to LSI technology. 
NT&T is stimulating R&D in industry through their tech¬ 
nical innovation such as electronic telephone exchange, data 
communication and picture transmission. The Electrotech¬ 
nical Laboratory, too, is promoting R&D in industry through 
governmental projects such as pattern recognition. 

In March 1976, VLSI (Very Large Scale Integration) 
Technology Research Association was established by the 
seven corporations in Japanese computer industry, such as 
Computer Development Laboratories, Inc., Fujitsu Limited, 
Hitachi, Ltd., Mitsubishi Electric Corporation, NEC-To- 
shiba Information Systems Inc., Nippon Electric Co. Ltd., 
and Tokyo Shibaura Electric Co. Ltd. 

This is based on the four year national project for devel¬ 
oping VLSI technology in cooperation with the Electro¬ 
technical Laboratory and NTT. R&D is carried out at three 
laboratories: Cooperative Laboratories, responsible for fun¬ 
damental technologies and Computer Development Labo¬ 
ratories, Inc. and Laboratories of NEC-Toshiba Information 
Systems Inc. both responsible for application technologies. 

The total budget for R&D is approximately 70,000 million 
Yen (Approx. $290 million) and R&D subjects are covering 


(1) microfabrication technology, (2) crystal technology, (3) 
design technology, (4) process technology, (5) test and eval¬ 
uation technology, and (6) device technology. 

The Science and Technology Agency, which is sponsoring 
the big national projects such as space science and technol¬ 
ogy and atomic energy, is exploring the possibility of de¬ 
veloping various subjects. The Agency picked up the subject 
“III-V Semiconductor for Functional Devices” and financed 
the Electrotechnical Laboratory and others for investigation 
and research on Gunn effect digital devices for computers. 

The Ministry of Education, Research and Culture is grant¬ 
ing financial aid for special research programs besides fi¬ 
nancing university for research done by individual staff or 
small group of staffs. 

One of the programs is “Surface Electronics,” new sur¬ 
face and thin film technologies of compound semiconduc¬ 
tors, and includes R&D of technology for insulator, mainly 
oxide film formation on the surface of GaAs wafers and of 
its application to surface passivation of GaAs devices and 
to fabrication of GaAs MOSFETs and integrated circuits. 
This was a three year project and terminated in March 1978. 

SILICON DEVICES 

Many technological advances have been made to improve 
high frequency characteristics of bipolar transistors and 
MOSFETs. Smart self-alignment technique was successfully 
applied to MOSFETs as DSA (Diffusion-Self-Aligned) and 
to bipolar transistors as SET (Stepped Electrode Transistor) 
to avoid critical mask alignment. Various types of DSA 
MOSFETs have been announced as a high speed, sub-mi- 
cron channel device with high drain avalanche breakdown 
voltage and without punch-through effect. Cut-off frequency 
of about seven GHz was reported even in the first experi¬ 
mental device by the Electrotechnical Laboratory. 1 

In order to raise the maximum oscillation frequency of 
bipolar transistors, very small inter-electrode gap between 
the base electrode and the emitter was realized by a self¬ 
alignment technique as shown in Figure 1. The Electrical 
Communication Laboratories called this transistor as SET 
and reported that the gap less than 0.4 /urn was achieved and 
the cut off frequency of | S 2 i e | 2 has reached to 8.4 GHz, 
although precise mask alignment whose tolerance was 
smaller than 0.3 jttm was not used. 2 
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SIT (Static Induction Transistor) was proposed and de¬ 
veloped in Tohoku University. This has a very short channel 
with high resistivity as illustrated in Figure 2 and triode- 
like current-voltage characteristics. The delay-time-power 
product is small and less than 0.01 pJ. 3 This device is useful 
also as a high output power transistor at microwave fre¬ 
quencies and Mitsubishi announced 20wSIT at 1GHz. 4 

GaAs DEVICES 

Gunn effect functional devices with Schottky control 
gates, developed in the University of Tokyo and other co¬ 
operating organizations, are expected to have very small 
delay time-power product 5 such as 0.001 pJ for Gao. 82 Ino. 1 sSb. 



N +n k emitter 
(a) 



Figure 2—The structures of new transistor types made by lateral p- 
diffusion. 

(a) bipolar transistor 

(b) static induction transistor 


This motivated the extensive study of material processing 
technology on III-V semiconductors in Japan. 

The special research program “Surface Electronics” stim¬ 
ulated the research of technology of oxide film formation on 
the surface of GaAs wafers in universities. A group at Hok¬ 
kaido University worked on anodic oxidation in the mixture 
of tartaric acid, glycol and water 6 and another group at the 
University of Tokyo worked on anodic oxidation in oxygen 
plasma. 7 In addition to R&D in universities, Matsushita de¬ 
veloped the technique of thermal oxidation of GaAs in re¬ 
duced ambient pressure 8 and Fujitu reported GaAs MOS- 
FETs with the maximum oscillation frequency of 13 GHz 
using plasma oxidation technique. 9 

HIGH SPEED LOGIC 

Logic integrated circuits with higher speed with more 
density of integration have been achieved by new circuit 
configuration and devices with better performance capabil¬ 
ity, supported by novel process technology. 

The Electrical Communication Laboratories developed 
the new technologies, called N±IPOS(N + Insulation by Ox¬ 
idized Porous Silicon) and E 2 IC (Elevated Electrode Inte¬ 
grated Circuit) process to minimize the isolation area and 
isolation capacitance and to make the size of bipolar tran¬ 
sistors smaller. Both N±IPOS and E 2 IC are self-aligning 
technologies and the former is associated with isolation layer 
formation and base diffusion and the latter is effective in 
electrode formation. 

Non-threshold logic proposed from the Electrical Com¬ 
munication Laboratories is a unique circuit configuration 
where bipolar transistors are operating in the active region 
only and never go to the saturation region. Therefore NTL 
IC fabricated by E 2 IC technology showed the very short 
delay time per gate of 85 psec and the very small delay time- 
power product of 0.075 pJ. 10 In 8 Bit ALU with 180 gates 
integrated in a chip of 1.6x 1.0 mm 2 the delay time was 260 
psec per gate. 11 

Using the configuration similar to I 2 L, SIT logic IC 
showed the power consumption of 7 /a w per gate and delay 
time of 30 nsec per gate and the smallest delay time-power 
product of 2 fJ. 12 

Improvement of I 2 L using dielectric isolation or vertical 
pnp injector has been done by the Electrical Communication 
Laboratories and Mitsubishi. The former reported the delay 
time of six nsec and the delay time-power product of 0.3 pJ 
for three collectors configuration 13 and the latter announced 
the delay time power product of 0.07 pJ. 14 

A new logic circuit configuration, Wired OR Wired AND 
Logic (W 2 L) was announced from OKI. Figure 3(a) illus¬ 
trates an inverter of W 2 L. Both A and B are the outputs due 
to an input, but the directions of the output current flow are 
opposite to each other as shown by the arrows. The output 
current of A terminal exists only when the logical output 
status is ‘H’ and of B only when the logical output status is 
“L.” Figure 3(b) shows a circuit configuration correspond¬ 
ing to Figure 3(a). The performance of W 2 L as shown in 
Table I is just intermediate between TTL and I 2 L. 15 







Semiconductor Technology in Japan 


1231 


<*) 



A 

B 


(b) 



Four bits ALU composed by n channel enhancement type 
DSA MOSFETs with n channel depletion type MOSFETs 
as load showed 2.9 nsec as the delay time per gate and 0.71 
mW as the power consumption per gate which was reported 
by NEC and the Electrotechnical Laboratory. 16 

Recently Mitsubishi announced DSA masterslice LSI for 
920 gates random logic, whose performance is listed in Table 
II. 17 

Punch through mode operation of a submicron gate MOS- 
FET is useful because of its inherent high speed performance 
and low output impedance. Fujitsu proposed to use this 
mode of operation for high speed logic. An experimental 13 
stage ring oscillator consisting of E/D inverters and buffer 
circuit by 1 /zm gate MOSFET (effective gate length of 0.5 


TABLE II—Feature of the DSA MOS Masterslice Chip 


Number of 

800 internal gate cells 


Circuits 

116 output gate cells 


Power Supplies 

V cc 5V single power supply 


Propagation Delay 

basic cell 

1.0 ns at 3.6 mW 

Time 

(minimum interconnection) 



basic cell 

(average interconnection 
of ALU) 

3.0 ns at 3.6 mW 

Output buffer 

TTL compatible 
t r 7 ns at 1 TTL load 
t f 5 ns at 1 TTL load 


Total power 

internal gate cells 

2.5 W 

dissipation 

output buffer circuits 

0.5 W 

Chip size 

7.68x7.88 mm 2 


Number of pins 

maximum 120 pins 



/xm) with 350 A gate oxide film showed 90 ps switching 
delay and an E/E inverter indicated the delay time-power 
product of 2 fj. 18 

Toshiba paid much effort for developing SOS devices and 
reported 1300 gates LSI by n channel Si gate, E/D config¬ 
uration. This LSI RFO (Register File O) is composed of 
250 bits static RAM and peripheral circuits. This chip size is 
4.14x4.47 mm 2 and the delay time per gate is 0.70 nsec and 
the access time is 65 nsec, the power consumption is 450 
mW and the delay time-power product per gate is 0.21 pJ. 19 
R&D of GaAs ICs were reported, too. 

Normally Off-Type GaAs MESFETs were used for high 
speed logic by Fujitsu. An experimental 13 stage ring oscil¬ 
lator showed the delay time of 280 psec and the delay time- 
power product of 74 0. 20 

Significant progress of Gunn effect functional devices has 
been made by the Electrotechnical Laboratory and Fujitsu. 
Fujitsu announced two bits shift register which could be 
operated at the clock rate of 1.6 GHz. 21 Recently 20 ps/gate 
Gunn-effect high speed carry finding device for 8 bit binary 
adder was released by the Electrotechnical Laboratory. 22 


MEMORY IC 


TABLE I 



T 2 L 

W 2 L 

I 2 L 

Delay Time 

ns/gate 

3-30 

3-30 

10-100 

Power Consumption 

mW/gate 

Threshold Voltage 

2-20 

1-10 

0.01-1 

V 

1.4 

0.7-1.4 

0.7 

Power Supply 

V 

5 

1.6-5 

0.8-5 

Load Drive 

good 

good 

poor 

Mask 

6 

6 

4 

Transistor Mode 

normal 

normal 

inverse 

Active Area Ratio 

/gate 

1 

0.4 

0.1 

/ALU 

1 

0.3 

0.2 


Significant reduction of power consumption in bipolar 
RAM was achieved. Fully static 4 K bipolar TTL RAM 
whose typical access time is 25 nsec and power consumption 
is as low as 350 mW has been developed by Hitachi. To 
reduce power consumption, stand-by current of a cell is 
decreased to 4 /tzA and it is increased by sinking current 
through lower word line at the instant when the cell is going 
to be selected. The fabrication process includes oxide iso¬ 
lation with double layer metallization and transistors with 
the emitter of 3 fx mx3 \x m in size have low parasitic capac¬ 
itances (C T s=0.13 pF, Grc=0.041 pF and C T e=0.034 pF). 

The chip size is 3.4 mmx6.3 mm. 23 Similar 4 K static 
bipolar TTL RAM was announced by NEC. The access time 
is 40 nsec and the power consumption is 500 mW. The size 
of a memory cell is 38 /xmx60 /zm and of a chip is 3.24 
mmx5.3 mm. Novel polycrystalline silicon self-aligned 
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(PSA) technique in combination with non-epi technology 
(diffused collector) and LOCOS process was used. 24 

The Electrical Communication Laboratories released a 64 
K bit MOS RAM whose access time is 200 nsec and power 
dissipation is 150 mW. MOSFETs whose effective channel 
length is 2 fxm, gate oxide thickness is 500 A, junction depth 
is 0.25 //,m and threshold voltage is 0.8 V at 1 V drain voltage 
are used. The technologies: As doped poly silicon gate, se¬ 
lective oxidation, Mo metallization, all ion-implantation, 
flowed phospho-silicate glass and fine pattern technology, 
2 fim pattern width, are incorporated. The cell size is 14 
/xmxl5 fim and the chip size is 6.1 mmx5.8 mm. 2 ? 

Clocked CMOS 4 K static RAM, measuring 4.7 mm 
square, has been developed by Toshiba. Conventional Si 
gate technology and fine patterning technology of 5 /zm were 
used and by narrowing the width of the boundary region, 
the cell area of 47.0 /zmx68.0 /zm was achieved. The access 
time is 200 nsec, and the operational power dissipation is 50 
mW at the cycle time of 1 //.sec and the stand-by power 
dissipation is 0.5 /zW. 26 Recently Toshiba reported new mul¬ 
tiplexed electrode per bit structure for a 64-K bit charge- 
coupled-device memory, where a merging serial register is 
combined with an ME/B array to make a practical and flex¬ 
ible CCD array. The memory operates typically at 5 M bits/ 
sec data rate, while a 512-bit test array is operated in less 
than 140 nsec transfer execution time. 27 

Non-volatile semiconductory memory has been devel¬ 
oped, too, as an erasable read only memory (EROM), an 
electrically alterable read only memory (EAROM), and a 
non-volatile random access memory (NVRAM). Two types 
of non-volatile semiconductor memory, which are a metal- 
insulator-oxide-silicon (MIOS) structure and a floating gate 
structure, were pursued. 28 As MIOS devices MAOS, 29 
MTa 2 05 -Al 2 03 OS 30 and MOSOS 31 were studied. As floating 
gate devices FAMOS is the first one, but it has disadvan¬ 
tages such that the controllability of stored charge amount 
in writing operation is not sufficient and the electron injec¬ 
tion ratio into the floating gate is rather small. These result 
in longer programming time. However these disadvantages 
have been removed by introducing avalanche injection with 
stacked gate structure (SAMOS), 32 which was developed by 
Toshiba. 

Recently Toshiba announced 2 K bit electrically alterable 
avalanche-injection type ROM with stacked-gate structure. 
The writing time is 20 //sec for a single transistor and is less 
than 5 sec for a fully decoded 2048-bit memory. Erasure of 
the memory is accomplished either by ultra-violet light ir¬ 
radiation onto the floating gate or by electric field emission 
of electrons from the floating gate to the control gate. 33 

The combination of hot electron injection and direct tun¬ 
neling injection in p-channel MNOS memory transistor was 
used to make a 1024 bit RAM, whose chip size is 3.60x3.61 
mm 2 , and read access time is 600 nsec and write cycle time 
is 10 /zsec-100 /xsec by Toshiba. 34 

Recently a single 5 V supply 16 K EPROM (Erasable Pro¬ 
grammable Read Only Memory) using DSA technique and 
self-aligned gate technology was announced by NEC. The 
access time is 300 nsec, the power consumption is 450 mW 
and the chip size is 3.6 mmx4.98 mm. 35 


CONCLUDING REMARKS 
It should be notified that: 

(1) self-alignment technique is successfully used to fab¬ 
ricate devices such as DSA and SET, and 

(2) much efforts are paid to develop new devices like SIT 
and Gunn effect digital devices. 

Of course steady improvement of conventional integrated 
circuits including devices used in them is going on and this 
is backed up with new process, technology and new circuit 
configuration. 
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The development of computers in Japan 

by OS AMU ISHII 

Electrotechnical Laboratory 
Tokyo,Japan 


INTRODUCTION 

Japan’s research and development of computers began in 
the 1940’s. However, Japanese production of commercial 
computers started a decade after the U.S. production did. 
Since then the computer industry has developed into the 
most rapidly growing industry in Japan. Thirty-nine thou¬ 
sand computers were installed as of the end of 1976. This is 
second in the world after the United States. 

It is difficult to cover all the computer industry and its 
R&D activities. The aim of this paper is to give a brief 
review of Japanese computer technology, with a short his¬ 
torical background, and a description of present status and 
some ongoing developments. 

DEVELOPMENT OF JAPANESE DIGITAL 

COMPUTERS 

The research on computers in Japan began at universities 
and national laboratories. Japan’s first digital computer, 
ETL Mark-1, a small scale experimental relay computer, 
was completed at the Electrotechnical Laboratory of the 
Japanese Government, in 1952, 1 Then the relay computers, 
FACOM 100 2 and ETL Mark-2 1 were constructed in 1954 
and 1955 respectively. These machines were used for 
Japan’s first computing services. 

The first stored program computer in Japan, called Fujic, 
which employed vacuum tubes and a mercury delay-line 
storage, was completed in 1956. 3 The research and devel¬ 
opment of transistor computers started almost at the same 
time and ETL Mark-3, a small experimental computer was 
also completed in 1956. 4 ETL Mark-4 , 5 which followed 
Mark-3, was constructed in the next year, and the technol¬ 
ogies developed were transferred to many types of com¬ 
mercial computers. 

In the same period, the parametron , 6 a logical element 
using the phenomenon called the parametric oscillation, was 
invented, and parametron computers PC-1 (1958), 7 and M-l 
( 1957)8 were constructed at the University of Tokyo and the 
Electrical Communication Laboratory (ECL) of Nippon Tel¬ 
egraph and Telephone Public Corporation (NTT). Several 
types of commercial computers later employed the technol¬ 
ogies developed for these pilot computers. 

Japanese companies began to manufacture and market 


computers around this time. In the 1960’s computers began 
to be introduced into various fields, but at the time Japan 
was said to be ten years behind the U.S. in technology and 
industrial management. Faced with this situation, the Japa¬ 
nese Government and computer industry realized that an 
extraordinary effort was required to close the gap. At that 
time the “National Research and Development Program” 
was established in 1966, and the “Super High-Performance 
Computer Development Project ” 9 was carried out by the 
Agency of Industrial Science and Technology of the Ministry 
of International Trade and Industry (MITI). The project was 
completed in 1971, and the technologies developed by the 
project were applied to several new commercial computer 
series in Japan. 

In the later half of the 1960’s, the era of third generation 
computers, domestic computer manufacturers announced 
their commercial series of computers. They were: FACOM 
230 Series (Fujitsu), HITAC 8000 Series (Hitachi), NEAC 
2200 Series (Nippon Electric, NEC), TOSBAC Series (To¬ 
shiba), MELCOM Series (Mitsubishi) and OKITAC Series 
(Oki). 

In 1968, NTT started the development of DIPS-1 (Den- 
denkosha Information Processing System-1 ), 10 which is a 
large scale computer designed for a nation-wide data com¬ 
munication service. DIPS-1 started offering the computing 
service in December 1973. 

Following DIPS-1, NTT developed in co-operation with 
the domestic mainframe manufacturers the DIPS-11 series 
which included three models. 

In 1972, under the promotion of the Japanese Govern¬ 
ment, the six domestic computer manufacturers were reor¬ 
ganized into three groups, and each group started the de¬ 
velopment of a new computer series. These series were 
called “M Series” (Fujitsu-Hitachi group), “ACOS Series” 
(NEC-Toshiba group) and “COSMO Series” (Mitsubishi- 
Oki group). The completion of all models of these series 
from large scale computers to small computers had been 
announced by the spring of 1977. 

COMPUTER INDUSTRY IN JAPAN 

The commercial production of electronic computers in 
Japan was started in the late 1950’s by electronics and 
telecommunications equipment manufacturers, as opposed 
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to business equipment manufacturers in the United States. 
Another point of difference between U.S. companies and 
Japanese companies is that the latter started with the 
production of solid-state computers skipping over vacuum 
tube systems. At that time, however, the computer industry 
was very new to the Japanese, not only from the point of 
view of technology but also from that of rental systems. The 
Japanese Government recognized the importance of this 
field of industry, and the Japan Electronic Computer Com¬ 
pany (JECC) was established with the support of MITI by 
domestic computer manufacturers in 1962 for the purpose 
of establishing a rental system in Japan. Through the 1960’s, 
the Japanese computer industry has progressed remarkably, 
especially in the later half of the decade, and application of 
computers has penetrated many sectors of Japanese society. 
Figure 1 shows the trend of computer production after 1965 
and the trends of peripherals, terminals and small business 
computers in Japan. The growth of the production of ter¬ 
minals reflects the rapid growth of on-line systems in various 
fields of computer applications, such as banking, seat reser¬ 
vations and inventory control systems. 

Table I shows the composite ratio of applications of on¬ 
line systems in Japan. 11 

Besides domestic computer manufacturers, there are 
many foreign manufacturers in the Japanese computer mar¬ 
ket. For instance, IBM and NCR have established their fully 
owned subsidiaries, and other companies are carrying out 
technical and sales activities in joint-ventures with Japanese 
firms. Others have sales offices in Japan. 



TABLE I.—Applications of On-Line Systems 


Fields 

Compostion 
(*) 

Finance, Securities, Insurance 

21.3 

Manufacturing, Sales, Stocks 

43.8 

Computing Service 

8.1 

Pollution Monitoring 

7.3 

Transportation* Traffic Control 

5.3 

Others 

14.2 

Total 

100.0 


The share of domestic and foreign products in the Japa¬ 
nese market are shown in Table II, as of the end of Decem¬ 
ber 1976. In terms of total system value, domestic produc¬ 
tion accounts for 56.8 percent, and imports for 43.2 percent. 
But for large scale systems, imports exceed domestic pro¬ 
duction. 

The market for small and very small computers is growing 
and they are penetrating the large computer market rapidly 
because of their cost performance. “Office computers” or 
“Office systems” (small business computers) are becoming 
feasible for small enterprises, and sales in 1977 are expected 
to reach 100 billion yen. 

Microcomputer applications are expected to increase 
markedly, not only for computers, but also for various types 
of peripheral controls and equipment controllers. Sales are 
expected to be 20 billion yen for the 1977 fiscal year. 

Japan has prudently liberalized capital transaction and 
trade regulations for the data processing industry. Although 
surrounded by controversy regulations for hardware were 
liberalized up to 100 percent in December 1975 and for 
software in April 1976. 

DEVELOPMENT OF 3.5 GENERATION COMPUTERS 

The three groups of the six domestic computer manufac¬ 
turers started to develop three new 3.5 generation computer 
series under MITI’s support in 1972. Since 1974, each group 
announced new computer models of their series one after 
another and all three completed their series by the spring of 
1977. 

The series are shown in Table III, with rough correspond¬ 
ence of the system scale between IBM System/370. Origi¬ 
nally the cost/performance of these series surpassed that of 
the IBM System/370. In 1977, however, IBM announced the 
303X processors, and cut the prices of the System/370, also 
the value of the yen increased, so the situation has changed. 

Technological trends adopted in the new series computers 
are: 

(1) Development of LSI and packaging technology 

(2) Stress on IBM compatibility 
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TABLE II.—General-Purpose Computer Installations in Japan as of the end of December 1976 (Source: MITI) 


System Scale* 

Domestic 

No. of Sets 

Systems 

Value** 

Imported 

No. of Sets 

Systems 

Value** 

Total 

No. of Sets 

Value** | 

! Large 

1,247 

703,634 

932 

761,885 

2,179 

j 

1,465,519 1 

Medium 

4,617 

456,967 

1,465 

183,938 

6,082 

640,904 ? 

Small 

8,235 

156,803 

2,788 

57,782 

11,023 

214,585 | 

Very Small 

10,985 

68,461 

8,658 

52,485 

19,643 

120,946 

Total 

25,084 

1,385,864 

13,843 

1 , 056,090 

38,927 

2,441,953 


* System scale Is classified by purchase price as follows: 

Large : More than 250 million yen 

Medium : 40- 250 million yen 

Small : 10 - *10 million yen 

Very small : Less than 10 million yen 

** Value; million yen, the conversion rates between Japanese Yen and 
U.S. Dollar are approximately 290:1 for 1976 


(3) Virtual storage and multi-processing 

(4) Considerations for data communication and data base 
oriented systems 

Table IV summarizes the characteristics of several rep¬ 
resentative new series computers (large scale models). 

Domestic mainframe manufacturers are also producing 
many kinds of peripherals including magnetic disk units sim¬ 
ilar to IBM’s. 

Many operating systems were developed. The operating 
systems for each series and model group are shown in Table 
V. The development of OS aimed at the realization of system 
functions and performance, which would equal or better 
IBM’s MVS SVS, OS/VSl or DOS/VS. The most important 
consideration in developing OS was compatibility with the 
operating systems which were being supported by computer 
manufacturers. 

NTT’s DIPS-1 was developed during the era of third gen¬ 
eration computers. DIPS-11, which is a 3.5 generation ver¬ 
sion of the DIPS-1, consists of Model 10, Model 20 and 
Model 30. Model 30, the largest system of the series, sur¬ 
passes the performance of the IBM System/370 Model 168. 
Table VI shows a summary of DIPS characteristics. These 
models have full compatibility with each other on software 
and peripheral equipment. 

Another trend in data processing is distributed processing, 
or computer networks. In 1974, IBM announced SNA (Sys¬ 
tem Network Architecture), and some extensions were in¬ 
troduced in 1976. After the introduction of SNA, most main¬ 
frame manufacturers announced their versions of network 
architecture. Japanese manufacturers for example an¬ 
nounced, ANSA (Advanced Network System Architecture, 


Toshiba), DINA (Distributed Information Processing Net¬ 
work Architecture, NEC), MNA (Multishare Network Ar¬ 
chitecture, Mitsubishi), MSNA (M Series Network Archi¬ 
tecture, Fujitsu-Hitachi), FNA (Fujitsu Network 
Architecture, Fujitsu), HNA (Hitachi Network Architec¬ 
ture, Hitachi) and DONA (Decentralized Open Network 
Architecture, Oki). NTT is now promoting DDX (Digital 
Data Exchange) and has also announced its own network 
architecture, DCNA (Data Communication Network Archi¬ 
tecture). These network architectures are not yet well es- 


TABLE III.—Correspondence of the New Computer Series with IBM 
Systems 


IBM System/370 

- 

M Series 

ACOS Series 

COSMO Series 

(Double to Triple 
of 168) 

190 

900 

Model 1, 2 


168 

180 

180 II 

800 

Model 1, 2 


158 

170 700 

160 600 

900 

145, 1U8 

160 II 500 

160 S 

. 

w 

w 

0 0 

0 0 

b- b- 

135, 138 

150 400 

500 

125 

i 4 o ; 300 

i ' 

115 1 130 j 200 

_i_i_ 

300 
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TABLE IV.—Summary of Representative New Series Computers (Large Scale) 


1 

-r 

M-190 

M-180 

ACOS-900 

COSMO-900 j 

! i 

! { 

> .j 

1 

; CPU | 

t t 

i \ 

Processors 5 

Performance ; 

Instructions 

Cache 

\ j 

1, 2 

155ns (ave.) 

193 

16 KB 

i 

j 

310 ns (ave.) 

195 

! 

16 KB 

1, 2, 3, 4 

3.4 MIPS 

301 

16 KB 

2 j 

0.6 MIPS 

186 | 

None 

\ i 

i 

\ | 

Capacity 

\ 

1 - 16 MB 

1 - 8MB 

1 - 16 MB 

0.5 - 4 MB ; 

■f 

t 

! Main 

Cycle time 

i 

480 ns 

400 ns 

600 ns 

550 ns i 

• Memory 

Width 

32 B 

8 B 

8 B 

8 B | 

i 

{ 

; 

Interleave 

2/4 way 

4 way 

2, 4, 8 way 

2 way j 

i 

i 

i 

1 Channel 
| 

Channels/CPU 

Throughput 

16 

20 MB/S 

16 

16 MB/S 

r - - - - 

18 

60 MB/S 



tablished. However, they will play an important role in fu¬ 
ture computer systems in Japan. 

RECENT TECHNOLOGIES AND PROJECTS 
System architecture 

An important goal of the development of the new com¬ 
puter series is realization of high OS efficiency for transac¬ 
tion processing and high level language oriented environ¬ 
ments. Existing computer systems need a lot of supervisor 


TABLE V.—Operating Systems Listed by Series and Group 


M Series 

ACOS Series 

COSMO Series 

190 

0SIV/F4 180 II 

160 

180 

VOS 3 170 

160 II 

900 

ACOS-6 800 

700 

600 

900 

UTS/VS 700 II 

700 

190 

0SIV/X8 180 11 
160 

140 

180 

VOS 2 170 

160 II 

150 

500 

acos-4 4oo 

300 

700 II 

RBM 

700 

160 II 
VOS 1 150 

180 II 

160 

0SIV/F2 l4o 

130 

180 

EDOS- 
MSO/M x1u 

160 

UPS 500 

ACOS-2 200 

DPS 300 


processing for multi-processing control, which causes an 
increase in system overhead. The new process oriented ar¬ 
chitecture and firmware implementation for multi-program¬ 
ming control reduces system overhead, and increases sys¬ 
tem throughputs. Also, by adoption of the high-level 
language oriented architecture, instruction steps for execu¬ 
tion of high level language statements were substantially 
reduced. Table VII shows the relative improvement of OS 
efficiency for the ACOS series. 12 


TABLE VI.—Summary of DIPS-1 and 11 


. 

1 

DIPS-11 


DIPS-1 


__ ____1 

(1975-76) 


(1971) 

Relative Performance 

Model 10 20 

1 1.5 

30 

3 

1 

• . . ■ „ 

Processors 

Max. 2 


Max. 4 


Instructions 

169 


160 

CPU 

■ • 

Cache 

8/16 KB 


8/16 KB 


Logic device 

MSI/LSI 


SSI/MSI 


Capacity max. 

8 MB, 16 MB 


16 MB 

Main 
; Memory 

Increment 

1 MB 


1 MB 


Device 

MOS'LSI 


Core 


Channel controllers 

Max. 2, Max. 

4 

Max. 6 

| Channel 

Channels/controller 

Max. 16 


Max. 16 

i_ 

Throughput/controller 

Max. 12 MB/S 


Max. 12 MB/S 
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TABLE VII.—Improvement of OS Efficiency 



System A 

System B 

I/O Overhead 



(Instruction step) 

1 

0.65 

Instruction per 



Function (COBOL) 

1 

0.54 


System A: Existing Architecture 

System B: Process and high-level 

language oriented architecture 

Virtual storage 

The concept of virtual storage became common after the 
IBM System/370 adopted it. The M, ACOS and COSMO 
series employ the similar virtual storage technology. The 
details of the control such as page or segment sizes are 
different in each series. In some of the new series machines, 
channel address translation mechanisms are implemented by 
hardware. This was made possible by the progress in sem¬ 
iconductor technology. 

Optional processors 

The array processors such as the CDC STAR 100 or the 
CRAY-I are provided in the new series for a high-speed 
arithmetic processing option. These processors are designed 
for ease of use, for example, Hitachi’s “IAP” allows stand¬ 
ard FORTRAN programming. 

LSI and package assembling 

Semiconductor LSI’s and LSI assembling techniques are 
the most important parts of modem computer hardware 
technology. 

For instance, the floor space, weight and power con¬ 
sumption of the main frame of DIPS-11 (1975-76) 13 were 
reduced one fourth to one third, one third to two thirds, and 
a half to two thirds respectively from those of its predeces¬ 
sor, DIPS-1 (1971). These improvements were mainly real¬ 
ized by the progress in LSI technology and assembly tech¬ 
niques. 

Typical LSI specifications used in the new series com¬ 
puters are shown in Table VIII. As an example, the logic 
LSI chip for the ACOS series, 14 is shown in Figure 2. In this 
case, the assembling has three distinct levels: 

(1) Semiconductor LSI chip: LCML (LSI oriented low 
energy CML), max. 200 gates per chip, 7 pico joule 
per gate. 


TABLE VIII.—Typical Specifications of LSI Used in M and ACOS Series 


\ It em 

1___:_ 

M Series 

ACOS Series 

1 

I 

Chip size 

if x itmm 

3.2 x 3.2mm 

l 

Gates/chip 

100 

Max. 200 

! Logic 

Delay time/gate 

0.7ns 

0.7ns 


Power disslpatlon/gate 

35mW 

lOmW 


Bits/chip 

it,096 

it,096 

i 

Access time 

100ns 

Max. 200ns 

j Memory 

Cycle time 

220ns 

Max. it00ns 

1 _ 

Power dissipatlon/bit 

0.12mW 

O.llmW 


(2) LSI package: 80mm x 80mm ceramic package, max. 
110 LSI chips (max. 3,500 gates) per package, 240 
connector pins. Forced air cooling. 

(3) High density printed board: 530mm x 390mm, 15 con¬ 
ductor layers printed board, 12 LSI packages (max. 
40,000 gates) per board. 

Figure 3 is a photograph showing the LSI chips on the 
package. 

Kanji input and output 

The Japanese language is usually written in a mixture of 
“Kanji” (ideographic) and “Kana” (phonetic) characters. 
There are now only 48 phonetic characters, but several 
thousand ideographic characters are used in Japan (2,000- 
3,000 characters are popular, sometimes up to 10,000 char¬ 
acters are needed). Because the Japanese language is not 



Figure 2—Logic LSI chip 
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Figure 3—Logic LSI chips on the package 


adaptable to a purely phonetic writing system “Kanji” are 
a most important medium for information exchange. How¬ 
ever, computer systems today are not powerful enough to 
handle “Kanji.” The main reason is lack of suitable input 
and output devices. 

For output devices, expensive character generators are 
needed because of the huge character sets and complicated 
form of each character. Figure 4'shows an example of Kanji 
printer output 15 which includes “Kanji,” “Kana” and some 
alpha-numeric characters represented by 24x24 dot matrix. 
Many types of memories have been used for Kanji font 
memories. However, the progress in LSI technology is re¬ 
ducing memory costs, so that one major problem, “Kanji” 
output, is expected to be solved in the near future. 

“Kanji” input is a more difficult problem than output 
because of its man-machine interactions. A conventional 
“Kanji” typewriter has a great many character keys and 
several shift keys because of its large character sets. Ac¬ 
cordingly, it is very difficult for even a skilled operator to 
use a “Kanji” typewriter. 

Table IX is a comparison of typical characteristics of the 
English alphabet and “Kanji” I/O devices. The input speed 
of the “Kanji” device is about one fifth that of the alphabet 
in characters per minute. However, one “Kanji” character 
sometimes corresponds to one word so that the input speed 
of total information would be better than one fifth. 

There are many types of “Kanji” selection mechanisms 


2 

4 

X 

2 4 K? 


r J 9 X 

3 

• 

7 

m m X 3 

♦ 7 m 

m 

m 

* 

2 

0 / & 





Figure 4 —An example 

of Kanji printer output 


for the input device. Table X summarizes trade off between 
input speed and training necessary for various “Kanji” input 
devices and systems. 

“Kanji” OCRs are being researched, in addition to OCRs 
for “Kana” and alpha-numeric characters now commer¬ 
cially in operation. 

The VLSI project 

Through the development of the 3,5 generation com¬ 
puters, it became clear that semiconductor technology, es¬ 
pecially ultra-high density integration techniques, will play 
a key role in the future computer industry. In view of this 
situation, domestic computer manufacturers are emphasiz¬ 
ing the “Very Large Scale Integration” (VLSI) project 
under MITI’s support and guidance. The VLSI technology 
research association was started by a joint effort of five 
domestic mainframe manufacturers, NTT, and the govern¬ 
ment’s ETL which provided technical guidance. The goal of 
this project is to develop the basic LSI and LSI production 
technologies for fourth generation computers. Production 
and process techniques for very precise (sub-micron) pat¬ 
terns using electron-beam, x-ray and conventional photo¬ 
lithography are being developed. From 1976 to 1979, a total 
of 70 billion yen is expected to be invested in this project. 

The development program of pattern information 

processing systems 

In today’s “computerized society,” computer utilization 
is becoming more and more varied, and increasingly higher 
efficiency and intelligence are expected. To cope with such 
requirements, it is desirable that future information systems 
advance into the area of pattern information processing. 

The Japanese Government started a research and devel- 


TABLE IX.—Comparison of Alphabet and Kanji I/O 


Character 

1 ; Set (C) 

Dot Matrix 
(D) 

Memory 
Capacity 
(C) x (D) 

Code 

Input Speed 

Price ($) 
(Typewriter) 

Alphabet (A) 

26 

5x7 

910 

1 Byte 

250 ch./min. 

100 






(50 words/min.) 


Kanji (K) 

4,000 

24 x 24 

2,304,000 

2 Bytes 

45 ch./min. 

500 

Ratio (K)/(A) 

150 

16 

2,500 

2 

0.18 

5 

_ 





(0.9) 
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TABLE X.—Comparison of Kanji Input Systems 



Input Speed 
Charact ers/Min• 

[Training* 

Kanji Keyboard with 
Multishift Keys 

o 

CO 

i 

o 

-=r 

| 

! 4 

Office Kanji Typewriter 

30 - 50 

3 

Kanji Tablet 

30 - 60 

2 

Mnemonic Code Input 

System 

60 -100 

5 

Interactive Input 

System 

. 

30 - 60 

1 

Kana-Kanji Conversion 
System 


1 


* 1: easy —5: hard 


opment program on “Pattern Information Processing Sys¬ 
tems (PIPS)” in July 1971. 

The research activities are: 

(1) Development of new materials and devices which may 
be applicable to pattern information processing. 

(2) Studies of subsystems for such visual and aural pat¬ 
terns as characters, speech, pictures and three dimen¬ 
sional objects, as well as for understanding natural 
language. 

(3) Development of information systems having such new 
capabilities as parallel processing, associative infor¬ 
mation retrieval, and inference, or learning. 

The activities in these research areas are closely related to 
each other. The final goal of the project is to construct a 
prototype system referred to as PIPS, which will be able to 
integrate the accomplishments attained in each area. 

The pilot models for character recognition, picture and 
three dimensional object recognition, and word voice rec¬ 
ognition, were demonstrated in July 1977. 

Besides these pattern recognition subsystems, a high-per¬ 
formance LSI microprocessor, 16 multi-microprocessor sys¬ 
tems, high-level language oriented machines, a magnetic 
bubble data base machine, and several other systems are 
under development. 

CONCLUSION 

Twenty years have passed since the commercial production 
of computers started in Japan. During this period the speed 
of technical development has been extremely rapid, and the 
applications of computers has increased greatly. It has af¬ 
fected almost every sector of society. The computer industry 
has become one of the most important industries in Japan. 
Now, there are new challenges facing the industry. In the 
next decade information processing systems which meet a 
wide variety of social needs must be developed. In these 
circumstances, technology which improves the cost/per¬ 
formance of computers will as always be sought after. But 


special applications related to traditional Japanese culture, 
mainly concerned with the language, will increase their role 
in Japanese society. 
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Remote data processing in Japan 

by KANJIRO KOSHI and KIMIO IBUKI 
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Tokyo,Japan 


INTRODUCTION 

This paper introduces the development process, some recent 
features and the perspectives of remote data processing in 
Japan. The development process shows that remote data 
processing in Japan has maintained a steady progress in 
number of systems and in its application. 

Recent features pertain to remote data processing in 
Japan, concerning recent application patterns. Also, a fore¬ 
cast is made of how the remote data processing in Japan 
will grow from the viewpoints of the boundary expansion as 
well as increases in technological capability. 

REMOTE DATA PROCESSING CHANGES AND 

PRESENT SITUATION 

A much more significant role will be played by remote 
data processing, in order both to cope with modem inno¬ 
vations and to meet requirements of sophisticated and com¬ 
plex business activities and management. Because remote 
data processing can process and transmit information. A 
long-term forecast has been announced wherein remote data 
processing will grow over 20 percent each year in the future. 

Computer usage in our country began in 1955 when the 
Nomura Security Co., Ltd. and Tokyo Stock Exchange in¬ 
troduced UNI VAC-120. The first remote data processing 
system was MARS, used by Japan National Railways, and 
JALCOM, used by Japan Air Lines. Both were seat reser¬ 
vation systems. Their operation started in 1964. There were 
40,000 general purpose computers installed as of March 
1977; 3,000 computers are used for remote data processing, 
showing a steady growth year by year. 

Remote data processing development background 

The background of computer usage development and re¬ 
mote data processing in Japan is outlined from social and 
economic, technological and legislative standpoints. 

Social and economic 

As the result of technological innovation, Japan has at¬ 
tained decent growth during the 1960’s and early 1970’s. 


Steel and petrochemical enterprises, electric power in¬ 
dustries etc. rationalized both productivity improvement and 
managerial efficiency by introducing new plants and tech¬ 
nology. The purpose was the extension of business scale 
and profit. 

This economic growth may be partly due to the existence 
of abundant, cheaper and high quality labor force. The in¬ 
crease in demand for labor force, caused by the economic 
development, brought about not only full employment but 
also both a shortage of manpower and wage hikes. As a 
consequence, enterprises were obliged to make efforts to¬ 
ward saving manpower. Hence, remote data processing be¬ 
came recognized as a basic and indispensable tool, because 
it has sophisticated data transmission and processing func¬ 
tions. 


Technological background 

Generally speaking, computer manufacturing enterprises 
as a newcomer should have totalized and highly advanced 
technological ability. Almost all computer manufacturers 
were at the same time communication equipment manufac¬ 
turers. 

In the 1960’s, NTT applied great efforts to provide na¬ 
tionwide direct-dialing telephone network through research¬ 
ing various technologies, such as electronic switching and 
multiplexing radio or carrier transmission by coaxial cables. 
Resulting from this research, NTT made various technolog¬ 
ical contributions and aids to communication equipment 
manufacturers. Furthermore, NTT vigorously promoted the 
DIPS development project during the latter half of the 1960’s 
and early 1970’s, with the cooperation of principal Japanese 
computer manufacturers. DIPS uses very large scale 3.5 
generation computers for NTT’s remote data processing use. 
Through this cooperation, NTT played a leading technolog¬ 
ical role and furnished significant impacts in pertinent tech¬ 
nologies and their advance. The Japanese government has 
also supported Japan’s computer industries since the early 
period, through various technological developments and re¬ 
search by government related laboratories on hardware and 
software. It was early recognized that computer industries 
will play a key role in changing the Japanese industrial 
structure into a knowledge intensive format. 
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TABLE I.—Computer Usage in Japan 


(As of March) 


Items - ' ____ 


1968 

1970 

1972 

1974 

1976 

1977 

No. of Sets 

nr 

3,558 

6,718 

■pPpSHf 

23,443 

35,305 

40,719 

General Purpose Computers 


— 

(37.9) 


(35.9) 

(17.3) 

(15.3) 

No. of Sets 

(B) 

— 

217 

476 

WMm 

1,871 

3,052 

Remote Data Processing 


— 

(56.1) 

(46.9) 

B 

(24.7) 

(63.1) 

Ratio of On-line 

Computers (B)/(A) 

(%) 

— 

3.2 

3.7 

4.4 

5.3 

7.5 


Remarks: () for percentage of growth rate compared with previous year. 


Legislative background 

Public telecommunication services providers vary accord¬ 
ing to countries. Japanese law dictates that these services 
should be provided only by NTT and KDD. 

There may also be various ways to operate remote data 
processing services, combined computers and telecommu¬ 
nications. In Japan, they are regarded as one of major tele¬ 
communication services and considered to be operated in 
principle in a competitive market. Furthermore, it was con¬ 
sidered desirable, for the progress of remote data processing 
technologies in Japan, that common carriers be permitted to 
operate those services. Thus, these common carriers were 
able to join that market under certain restrictions. In 1971, 
the Public Telecommunications Law, mainly for telephone 
and telegraph, was amended to legalize the provision of 
remote data processing. This amendment enabled not only 
common use of leased lines by a group of companies or the 
operation of remote data processing by private companies, 
but also their operation by NTT and KDD. It significantly 
contributed to the development of remote data processing 
in Japan. 

Since then, remote data processing systems are provided 
in three ways. One is implementation by users. Second is 
implementation and vending by NTT or KDD. Last is im¬ 
plementation and vending by private companies. Each con¬ 
tributes to the development of remote data processing in 
cooperation, making the most of their individual character¬ 
istics. 


Remote data processing changes and present situation 

Computer usage in Japan 

Table I shows the change in the number of general purpose 
computers and that of computers for remote data processing 
use. It can be seen that, during the past decade, each number 
has increased 11.4 times and 14 times, respectively. The 
general purpose computers growth rate has been 32 percent 
each year and that of computer for remote data processing 
use has been annually 46 percent during the same period. 
So, the remote data processing growth rate is larger than 
that of general purpose computers. This means steady prog¬ 
ress in remote data processing, as is also shown in increasing 


ratio of on-line computers. This trend is very prominent, 
especially during these past few years. 

Change in number of remote data processing systems 

Table II shows the change in the number of remote data 
processing systems. As shown by the table, the number of 
systems now is 25 times larger than in 1969. The average 
growth rate is nearly 50 percent annually. 

Remote data processing application areas 

Remote data processing application areas are shown in 
Table III. The table shows that the proportion has hardly 
changed. About 65 percent of all remote data processing 
systems are occupied by finance, insurance, securities sys¬ 
tems and production, inventory control or sales management 
systems. Therefore, remote data processing may be sum¬ 
marized as mainly used to process massive business admin¬ 
istration created data in order to increase business efficiency 
or rationalize manpower. Furthermore, it will be clear that 
new type systems are growing because pollution control 
systems have increased about seven times and traffic control 
systems have increased four times in number. 

SOME RECENT REMOTE DATA PROCESSING 

USAGE FEATURES 

Upgrading initial systems 

In Japan, banking organizations have been the major users 
of remote data processing since early years. Banking type 
systems still occupy a larger proportion of all the systems. 
The initial purpose of these systems was simply to improve 
business efficiency. However, they have been revised and 


TABLE II.—Changes in Number of Remote Data Processing Systems 

(As of March) 




No. of Remote Data 
Processing Systems 

84 

1971 

213 

1973 

490 

1975 

1,190 

1977 

2,079 

Annual Growth Rate (%) 

— 

53 

48 

55 

39 
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TABLE III.—Remote Data Processing Application Areas 


(As of March) 


Items -- ^ 

1972 

1973 

1974 

1975 

1976 

1977 

Finance -Insurance -Securities 

24.5% 

26.1% 

22.7% 

18.8% 

21.3% 

17.6% 


(81) 

(128) 

(174) 

(224) 

(319) 

(365) 

Production-Inventory Control and 

39.1 

38.8 

42.1 

45.5 

43.8 

49.8 

Sales Management 

(129) 

(190) 

(322) 

(541) 

(658) 

(1,036) 

Data Processing 

11.5 

9.8 

9.1 

8.0 

8.1 

7.4 


(38) 

(48) 

(70) 

(95) 

(122) 

(153) 

Pollution Control 

5.5 

7.6 

8.2 

6.7 

7.3 

5.9 


(18) 

(37) 

(63) 

(80) 

(109) 

(123) 

Traffic Control 

5.2 

5.9 

5.2 

4.6 

4.1 

3.6 


(17) 

(29) 

(40) 

(54) 

(61) 

(74) 

Commodities Transportation 

2.4 

2.0 


1.3 

1.2 

1.8 

Control 

(8) 

(10) 


(16) 

(18) 

(37) 

Other Use 

11.8 

9.7 

11.1 

15.1 

14.2 

13.9 


(39) 

(43) 

(85) 

(180) 

(214) 

(291) 

Total 

100.0 

100.0 

100.0 

100..0 

100.0 

100.0 


(330) 

(490) 

(766) 

(1,190) 

(1,501) 

(2,079) 


Remarks: () number of systems. 


converted into second or third generation for totalizing or 
extension of application areas. Thus, banking type systems 
are playing a vital role in our economical activities. A na¬ 
tionwide banking system was installed in 1973. This system 
is a message switching system connected to 7,600 offices of 
87 banks. It handles communications pertaining to and set¬ 
tlements of inter-bank domestic exchange transactions. As 
represented by this example, such systems that cover in¬ 
dependent companies and form computer networks tend to 
increase. 

Catching up on computerization efforts 

Minor enterprises in Japan are now on their way to com¬ 
puterization. Larger enterprises mostly completed comput¬ 
erization in the early period and have replaced their former 
systems by on-line systems. In recent years, small business 
computers and remote information systems have become 
popular among minor class companies as suitable tools for 
their business scale. There have also appeared systems that 
are shared among companies engaged in the same business 
field. 

Such efficient and totalized utilization will become such 
that small scale business computers, of the stand alone type 
possessed by minor companies, will be linked to a large 
scale computer system. Thus, remote information services 
will play a complementary role as data processing devices 
for minor class enterprises. 

Increase in systems close to our life 

The improvement of social and national welfare is being 
required more and more as Japanese people deem the quality 
of their lives more important than economical affluence. 


Therefore, they have begun to pay an attention to systems 
which satisfy those requirements. NTT has already imple¬ 
mented Emergency Medical Information System, Environ¬ 
ment-Pollution Information System, En-route Radar Data 
Processing System and Foodstuff-Market Information Sys¬ 
tem. Because each pattern is new and different from the 
others, it is true that there are many problems to be solved 
in the field of technologies, hardware and software and sys¬ 
tem operation. 

Remote data processing perspectives 

Remote data processing perspectives can be viewed from 
three viewpoints. They are the extension of boundaries, 
numerical increase and technological development. 

Enterprise class systems 

Computerization in enterprises and industries has contin¬ 
ued growing by nearly 30 percent each year during the past 
decade. Even now it keeps growing steadily. Remote data 
processing usage is expected to increase with increasing 
demand for inter-enterprise type of systems besides the in¬ 
crease of intra-enterprise type of systems. Enterprises tend 
to implement remote data processing systems that can han¬ 
dle settling accounts, message exchange or business calcu¬ 
lation among business related enterprises, a certain group of 
enterprises or those in the same business. 

On the other hand it will be probable that cheaper POS 
terminals will become more popular among shops or super¬ 
markets. In this case, such systems as are combined with 
POS terminals will appear. This kind of system will have 
great impact on society. 

With the increase in demand for inter-enterprise systems, 
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it will be more and more necessary to implement the public 
data network as a common tool. Utilization of shared type 
database will also be more frequent among enterprises in 
the same field or by a certain group of enterprises. 

Systems related to social welfare and national life 

There are many fields left to be computerized and many 
problems to be solved. Examples of underdeveloped fields 
are pollution control, environment protection, medical treat¬ 
ment, education, human and commodities transportation 
control, disaster countermeasures, commerce and improve¬ 
ment of local life standard. NTT has already researched and 
implemented such systems as Automated Meteorological 
Data Acquisition System, Environment-Pollution Informa¬ 
tion System, Emergency Medical Information System and 
En-route Radar Data Processing System. Although each of 
them is obtaining good results, their portion is still very 
small. Therefore, this class of remote data processing usage 
today can be said to still be in its infancy. However, con¬ 
sidering that many remote data processing systems related 
to this class are being energetically discussed now, intensive 
demand will appear before long. (cf. Table IV) 

Many systems of this class would take a certain time to 
become popular with the public. The reason is that the cost 


and resulting benefits of a system of this class don’t balance 
now or that the present demand is not strong enough. It is 
generally conceived that governmental subsidization or pa¬ 
tronage would be an incentive to the demands of this kind. 

Strong similarity exists among individual systems used by 
local government, in education or in medical treatment. If 
the cost of system development is reduced far enough and 
the cost and resulting benefits balance by the standardized 
implementation or sharing software, this would be good 
stimulation for the realization of this class of systems. 

On the other hand, a system of this class has the aspect 
that massive data is stored and that it often handles infor¬ 
mation service or information retrieval services. These char¬ 
acteristics would signify that database oriented systems in¬ 
crease in number. From the standpoint that the systems of 
this class tend to be used publicly, the security or integrity 
of data has especially to be assessed carefully. 

Personal usage class systems 

There are two examples of this class in operation in Japan. 
One is the system for calculation by pushbutton telephone, 
called DIALS. The other is the seat reservation system for 
the “bullet” train using the pushbutton telephone. This ser¬ 
vice is provided by Japan National Railways. Based on the 


TABLE IV.—Examples of Systems Now Being Discussed 


Social 

Needs 

Areas 

Social 

Activities 

Health and 
Social 
Security- 
Maintenance 

Social 

Utility 
Improvement 

Quality 
of Life 
Improvement 

Intellectual 

Improvement 

System Examples 


o 

o 


o 


Emergency Medical Information 

Medical 

o 

o 


o 


Regional and Remote Health Care 

Treatment 


o 



0 

Medical Bibliographic Information Retrieval 



o 


o 


Hospital Information 

Human and 

o 

o 




Aviation Control 

Commodities 

o 

o 


o 


Regional Traffic Control 

Transportation 

o 

o 


o 


Total Motor Vehecle Control 

Control 

o 





Air Cargo Clearance 


o 





Port Information 

Pollution Control 


o 




Air Pollution Control 

and Environment 


o 




Water Pollution Control 

Protection 


o 




Environment Information 

Disaster 

o 

o 




Automated Meteorological Data Acquisition 

Countermeasures 

o 

o 




Emergency Disaster Information 

Education 



o 

o 

o 

C A I 





o 

o 

Bibliographic Information Retrieval 

Commerce 

o 


o 

o 


Foodstuff Market Information 


o 





Petroleum Distribution Information 

MMMNH 

o 


o 

o 

o 

Home Video Information 

HHHHI1 




o 


Municipal Hotel Reservation 
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number of systems of this class implemented, the present 
situation is still in its infancy. 

To popularize remote data processing for personal use, 
service providers need to stimulate its demand as a first 
step. Popularization of this class mostly depends upon 
whether such service can be provided that is attractive and 
easy to access at a cheap price. 

It is very likely that home computers or on-line terminals 
will become as popular as telephones or electronic calcula¬ 
tors with the rapid price reduction for microcomputers or 
minicomputers. In this situation, the second step is expected 
to be demands for various services appearing from the users. 
In this stage, remote data processing may be said to be 
completely popular enough. In the near future, automated 
telemetering service, reservation service and information 
service are considered sure to appear. Automated teleme¬ 
tering service is now being put to the test in the suburbs of 
Tokyo. 

Basic technologies 

Computer network 

As remote data processing service extends its boundaries 
and increases in number, steady progress in computer net¬ 
work technologies is expected that will enable sharing hard¬ 
ware, software and database. In case of centralized com¬ 
puter systems, the expansion of its size does not necessarily 
bring economical or functional scale merits. This is consid¬ 
ered to be the principal reason why a distributed system has 
appeared. Besides this, three other reasons can be pointed 
out. One is that the plan for digital switched data network 
has appeared from the network technology side. The second 
one is that the network architecture has been proposed from 
the computer technology side. The last one is that pertinent 
device technology has rapidly advanced. 

In Japan, NTT plans to implement both a circuit switched 
network and a packet switched network in early 1979 as a 
public data network for computer connection. For network 
architecture, NTT also plans “Data Communication Net¬ 
work Architecture (DCNA).” It defines the interface con¬ 
ditions and protocols among the elements of remote data 
processing systems. The main aims are the improvement of 
flexibility in systems, network transparency, decentralized 
processing and linkage of independent computer systems. 
With DCNA, it will be easier to connect computer systems 
made by different manufacturers. 

Remote information services 

There are several dozens of companies in Japan that offer 
remote information services. Usage and technology are ex¬ 
pected to change as follows. 

(a) The present usage may be summarized as the sharing 
of computer power at low cost. From now on, the 
usage level will be more sophisticated, utilizing the 
highly developed functions of remote information ser¬ 


vices or applications such as forecast, simulation and 
so on. 

(b) It will be more popular to share and utilize high-quality 
databases or efficient applications that some other 
users have developed. 

(c) With the rapid popularization of microcomputers and 
small scale business computers, users will tend to 
utilize both small scale business computers and remote 
information services, depending on each merit. They 
will use small scale ones for simple business data 
processing or patterned data processing. They will use 
remote information services mainly for sophisticated 
and large scale data processing capability. 

Database 

Although the present utilization of databases is not so 
popular in Japan, utilization will be prompted more here¬ 
after. The following are expected for database utilization 
and related technologies. 

(a) Enterprise will introduce database systems for busi¬ 
ness use, because the uniformity of information will 
be recognized as more important to make business 
activities more efficient. 

(b) Governmental systems often store massive amounts 
of data. Various merits would be derived from the 
DBMS adoption. A steady increase is expected on 
systems in numbers that adopt DBMS if cost-benefit 
is sufficiently improved, with the technological ad¬ 
vance of mass storage devices. 

(c) Large scale systems often store bulk information and 
have nationwide spread. In this case, distributed da¬ 
tabase will often be preferable when considering 
avoiding software complexity, reliability improve¬ 
ment, peak traffic smoothing and reduction in com¬ 
munication cost. For this reason, distributed type of 
database will be popular, especially among large scale 
governmental systems. 

(d) Some remote information service offerers will appear 
trying to offer specified kinds of databases and appli¬ 
cations. 

EPILOG 

Computer technology was born in the U.S. and assisted 
greatly in man k ind’s dream to send people to the moon. 
This technology was transplanted to Japanese soil carefully 
nurtured, sprouted and rooted. It also withstood the hard¬ 
ship of the winter caused by the oil crisis and is now growing 
steadily. Japan has elevated the standard of remote data 
processing technologies, combined with telecommunication 
technologies, through persistent efforts and adding various 
original ideas. Some remote data processing systems may 
even be said to be unprecedented and to have no similar 
example in other countries. Remote data processing is ex¬ 
pected to have great impacts on society, economy and way 
of life. 
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INTRODUCTION 

It was in 1964 that the Shinkansen made its debut between 
Tokyo and Shin Osaka. It was extended westward to Oka¬ 
yama in March 1972 and then on to Hakata in March 1975. 
It is now a more than 1,000-km long traffic artery of Japan 
and is functioning smoothly. 

Constructed as a safety-performance high-speed means of 
passenger transport en masse in great comfort, the Shinkan¬ 
sen has proven its worthiness, playing a vital role in the 
economic and social growth of the nation. To have it play 
its part, the latest technical achievements are introduced to 
produce its rolling stock, track, electric facilities, etc. and 
control the operation of these aided by computer. It may be 
said that the Shinkansen is a typical modern railway oper¬ 
ating under a total system. 

In this paper, the conditions needed of the train operation 
control system for a high-speed mass-transport railway and 
how to meet these conditions will be explained. 

DEVELOPMENT OF TRAIN OPERATION CONTROL 

SYSTEM 

The Shinkansen calls for a train operation control system, 
vastly and basically different from that for conventional 
railways. One of the conditions attributable to the difference 
is the incomparable high speed. Here the conventional way 
of the driver operating his train by keeping his eyes on the 
wayside signal indications would never work. For this rea¬ 
son we employed the cab signal system together with, what 
is called, ATC (Automatic Train Control). To meet another 
condition of mass transport, there is CTC (Centralized 
Traffic Control) at work, though this is not entirely new. 
However, when it comes to a high-speed mass transport, 
like the Shinkansen, CTC has been proved to be an effective 
means of smooth dispatching and transportation. With the 
apparatus for these systems and the conventional inter¬ 
locking device, the Tokaido Shinkansen was opened in 1964. 

As the number of trains increased after 10 years, the CTC 


alone could not help the dispatches owing to a flood of 
information coming into the control center; therefore, CTC 
had to be backed up with computer. That is to have the 
computer do as much as possible to control train operation, 
collect data, classify these and do the transmission and to 
have the dispatchers devoting themselves to the job of mak¬ 
ing judgments by using the computer at will. To meet this 
demand, COMTRAC (Computer-aided Traffic Control Sys¬ 
tem) was introduced. 

OUTSTANDING FEATURES OF TRAIN OPERATION 

CONTROL SYSTEM 

In order to operate high-speed trains in large numbers, 
safely, precisely and efficiently, the basic requirement for 
train operation control is to have all the elements of the 
system work together smoothly under prescribed plans (train 
diagram and working schedule). 

With the trains that run on given rails, unlike any other 
modes of transport, it is clear that- high efficiency can be 
obtained when they are operated under a meticulously 
planned diagram, instead of in a haphazard way. Even when 
confusion takes place for some reason, it is possible for all 
the elements of the control system to get down to work as 
a whole with the single purpose of “restoring the prescribed 
plan.” One of the outstanding features of train operation is 
that it works as all planned out. 

When train operation is disrupted seriously by a train 
accident or a natural disaster, it also is possible under the 
control system to command all its elements to partially 
change the plan. 

Unlike other industries in general, train operation is a 
commodity that cannot be kept in stock. It has to be pro¬ 
vided to suit the demand of the passengers. Overall revision 
of train diagram once in two to three years and quarterly 
minor revisions, therefore, are taken in as a matter of 
course. The train operation system with these features is 
shown in Figure 1, divided into three functions of Planning, 
Doing & Seeing. 
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Figure 1—Schematic drawing of train operation control system 


COMTRAC 

Basic concept of reliability design 

Various functions shown in Figure 1, closely link them¬ 
selves together to do the functions. When the reliability, 
data quality, massiveness of data handled and response time 
are arranged, we have Figure 2 (COMTRAC Reliability 
Characteristics). The whole Operation Control System is 
seen here in a stairway structure, the lower the step, the 
higher in reliability and the more real time-like. The higher 
the step, on the contrary, the higher in data accumulation 
and in judgment complication. Another point to be noted is 
that the Operation Control System is so made up that once 
the upper-step function fails out, a lower step fail-back part 
of the function has failed out. 

With this adapted, the stairway structure is taken in the 
COMTRAC System. As higher reliability and response are 
demanded of lower functions and as higher-step functions 
demand more data accumulation and judgment complica¬ 
tion, the functions of the system are divided into two groups 
according to the evaluation criteria. That is, into one group 
of those functions that strongly demand reliability and re¬ 
sponsiveness to be accommodated in the route control sys¬ 
tem and the other group of functions demanding all sorts of 
data and others higher-level judgments in the operation ad- 



Reliability 

Figure 2—Reliability characteristics in COMTRAC 

justment system. As upper level functions call for more and 
more of those data that the computer is incapable of handling 
and more and more of human judgment, a man-machine 
system is needed, leaving the lower-level functions calling 
for less human intervention to be performed by an automatic 
system. 

COMTRAC functions 

A new train traffic control system with computer control 
is schematically shown in Figure 3. Among many functions 
necessary for the traffic control, COMTRAC is so designed 
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Figure A —Computer configuration of COMTRAC phase-2 


that it can perform the following functions consisted of five 
subsystems. 

1. Planning Subsystem that compiles the plan to imple¬ 
ment train and rolling stock turnarounds in accord with 
the basic and extraordinary plans for train operation. 

2. Dispatching Subsystem that monitors whether trains 
are being operated normally according to the train op¬ 
eration diagram prepared for the day by the Planning 
Subsystem. On detecting a diagram confusion, the 
computer automatically renders judgment, if the case 
is a simple one. If the confusion is serious, due to a 
train accident, for instance, the train diagram for four 
to five hours is predicated by simulation and the system 
prepares a train diagram thereafter and indicates it on 
GD (Graphic Display). Diagram modifications as de¬ 
cided through the computer are automatically trans¬ 
mitted from time to time to places concerned. 

3. Route Control Subsystem that traces each train as dia¬ 
grammed by the Planning and the Dispatching Subsys¬ 
tems and automatically sets its route. It also monitors 
the signals in all the stations. 

4. The Subsystem to provide data for station announce¬ 
ments. Automation of route control enables the system 
to detect train positions and train delays, and the in¬ 
formation is made use of in making station announce¬ 
ments. The train-to-leave signs on the platforms are 
automatically controlled, too. 

5. Statistics Analyzing Subsystem that prepares all sorts 
of reports on trains operated and compiles reference 
materials for administration. 


TABLE I.—Types of Passenger Train on Increase (Stopping Station 
Pattern) 


At the time of 
opening between 
Tokyo & Shin- 
Osaka 

Between Shin 

Osaka down 

to Okayama 

Between Okayama 

down to Hakata 

2 types 

5 types 

9 types 


TABLE II.—Route-Setting Automation Checking Growth in the Number of 
Dispatchers 



Before 

Ph-1 when 
extended 
to Okayama 

Ph-2 when 
extended 
to Hakata 

A: Train dispatchers 

43 

52 

51 

B: Electric railcar train 
dispatchers 

14 

15 

28 

C: Section length (km) 

520 

680 

1,060 

A + B 

C 

0.110 

0,099 



Concept of system composition 

In producing a computerized system for the functions 
explained earlier, the degree of reliability and the response 
time demanded of each function should be clarified and the 
unique characteristic of each function need be taken fully 
into consideration. 

The route control system that is directly concerned with 
train operation is completely a process-control system, set¬ 
ting train routes by the data gathered in the Center by CTC 
and in accord with the train diagram. It being the last output 
part of COMTRAC, the highest reliability is demanded of it 
and the data handled here also must be high in reliability. 

The Dispatching System detects diagram confusion by 
tracing each train with the data provided by the Route Con¬ 
trol System and then it really starts displaying its function. 
As the automatic judgment portion of it is considered to be 
similar to the monitoring function in the case of train oper¬ 
ation, we thought it be better performed under the Route 
Control System and set one minute as the target for the 
response time for the diagram simulation function. As for 
the data transmission function, we thought it hardly matters 
if acknowledgment of its receipt is made of a change in the 
plan within several minutes at the worst after the change is 
decided upon. 

The Dispatching System is to be made up as man-machine 
system as stated before, with man-machine interface cen¬ 
tering around GD (Graphic Display) for complicated judg¬ 
ment simulation and around CD (Character Display) for 
simple item simulation. It is made up as a multicomputer 
system with two divisions, one a highly reliable system, 
centering around route control and the other a data proc¬ 
essing system centering around train operation adjustment 
with man-machine equipment included and the fail-back idea 
partly taken in, as is shown in Figure 4. 


TABLE III.—Number of Trains and Lever-Handling Frequencies 



Before 

Ph-1 when 
extended 
to Okayama 

Ph-2 when 
extended 
to Hakata 

No.of trains a day 

170 

231 

275 

Lever-handling frequency 
a day 

4,200 

7,000 

10,500 
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TABLE IV.—Availability 



Apr; 

'76 

May 

Jun. 

Jul. 

Aug. 

Sep. 

Oct. 

Nov. 

Dec. " 

Jan. 

’77 

Feb. 

Mar. 

Average 

System 

Availability 

100 

100 

100 

99.93 

100 

100 

100 

100 

100 

100 

100 

100 

99-99 

Dual 

Availability 

99.89 

100 

100 

99.85 

99.96 

99.84 

100 

100 

99-90 

99.69 

99.97 

99.92 

99-92 

No.of system 
down cases 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

H 


(Remark: The one case of system-down was due to software bug) 


With this make-up, the Route Control System is as reliable 
as CTC with the target of 99.99 percent set for the working 
efficiency, and the Train Dispatching System, also with the 
target of 99.9 percent, downtime within 10 minutes. 

To attain these targets, a 2-out-of-3 computer system was 
developed for the Route Control System, and a DSC (Dual 
System Controller) for the Train Operation Control. 

As the Train Dispatching System handles a great number 
of data for many hours, two large-sized computers are in¬ 
troduced for duplex operation. 

EFFECTS OF SYSTEM INTRODUCTION & 

WORKING COEFFICIENCY 

In carrying on high-speed mass transport the best way to 
secure efficiency, safety and preciseness is to perform all 
the jobs concerned under single command, based on the 
diagram prepared by the Train Diagram Planning System. 

Thus, when a computer system takes care of diagram 
planning with infallible data, the following effects can be 
expected: 

(1) Infallible train control is possible (Prevention of route 
missetting)—Prevention of such accidents as train col¬ 
lision to preserve safety is ensured by ATC and inter¬ 
locking device on the spot. However, missetting of 
routes for high-speed mass transporting trains gives 
rise to train diagram confusion; hence missetting is 
regarded as a serious accident in train operation ad¬ 
ministration. Although there are no detailed records 
of route missetting before COMTRAC was intro¬ 
duced, such cases appear to have taken place several 
times a month. For the large number of various types 
of trains operating the Shinkansen today, with the 
stopping station pattern as it is, it surely would be 
impossible to set routes by human hand without COM¬ 
TRAC at work, even if the number of dispatchers 
were increased. After the introduction of COMTRAC, 
there were three cases of route missetting—two cases, 


programming misses, and one, an input miss by dis¬ 
patcher due to man-machine trouble. 

(2) Control of trains diversifying in type to meet the de¬ 
mand is possible—Users’ demands on the Shinkansen, 
the main traffic artery in Japan and long in operating 
section, change from season to season and diverse 
types of train are operated to meet their demands. 
Introduction of COMTRAC has thus brought forth a 
complicated transport form of operating diverse types 
of train to meet the users’ demands. In Table I, the 
diversification of train types is shown. 

(3) The growing number of dispatchers checked by au¬ 
tomation of route setting—The number of dispatchers 
before and after COMTRAC is shown in Table II. 
Under Ph-1 System, data transmission and rolling 
stock turnaround control were not performed but the 
system had the effect of checking the growing number 
of dispatchers. The systematization also had a notable 
effect in changing the dispatchers’ work from lever 
handling for route setting to their primary job (train 
operation adjustment, for instance). With the intro¬ 
duction of Ph-2 System, data transmission and rolling 
stock turnaround control were performed and the ef¬ 
fect of checking the growing number of dispatchers 
has grown all the more. 

(4) It is possible to cope with the demand for high-speed 
mass transport—The growth of the number of trains 
and the route setting frequencies are shown in Table 
III. The route setting by lever operation before COM¬ 
TRAC averaged once in every one to two minutes. At 
the time of opening the Shin Osaka-Okayama Section 
it grew to once in every 30 sec to one minute and then 
to once in every 10 sec to 20 sec when the Okayama- 
Hakata Section was opened. It is impossible for man 
to handle the lever for route setting at such a fre¬ 
quency. The System is presently working nicely with 
the target reliability value attained. There has been 
only one case of system-down during the year from 
April 1976 to March 1977 and this was due to software 
logic bug. (Refer to Table III.) 
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FUTURE OF COMTRAC 

JNR has two other Shinkansen lines, Tohoku and Joetsu, 
scheduled for opening within several years and is planning 
to form a nationwide Shinkansen network. COMTRAC 
will then be indispensable, as more and more diverse types 
of high-speed mass transporting trains have to be operated 
within the network. To be ready for it, COMTRAC will have 


to expand its scope of control, improve its functions to the 
infallible extent and grow not only into a train operation 
control system but also into a traffic control system, as well 
as to have it linked to MARS (Seat reservation system), 
Rolling Stock Control System and Management Information 
System, so that it can provide service more commensurate 
with the transport demands. 
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INTRODUCTION 

Since the first introduction of the microprocessor, various 
kinds of microprocessors have been developed. There have 
been remarkable achievements both in the level of integra¬ 
tion and processing speed. However, most microprocessors 
developed to date are oriented to device control applications 
or low class minicomputer replacement. It is true that some 
bit-sliced microprocessors can operate at very high speeds, 
but they are neither one-chip, nor do they have much func¬ 
tional capability. 

On the other hand, with the current state of the art in 
semiconductor technology, it seems possible to integrate 
considerable amounts of computing power in a chip and at 
the same time to achieve high speed operation comparable 
to existing discrete processors. 

The Electrotechnical Laboratory and Tokyo Shibaura 
Electric Co. (Toshiba) R&D center are engaged in a research 
effort in this area and have developed a new high-perform¬ 
ance universal computing element—PULCE. This paper de¬ 
scribes its architectural design, 1,2 as well as the semicon¬ 
ductor technology develped for its implementation. 

BASIC DESIGN 

Initial target of semiconductor technology 

At the beginning of the project, future trends of semicon¬ 
ductor technology were thoroughly investigated by evalu¬ 
ating pioneering experimental results 3 and the annual rate of 
improvement in the performance of MOS-LSI’s. 

The targets for 1977-1978 chip production were fixed in 
1972 by the Electrotechnical Laboratory. On the basis of 
extrapolated chip size, line width in photoengraving tech- 


* This work was done as part of the PIPS (Pattern Information Processing 
System) Project, an ongoing research program funded by the Japanese Gov¬ 
ernment. 


nology and annual rate of increase of integration of MOS- 
LSI’s, the number of gates per chip was estimated as more 
than 5000 but less than 10,000. Assuming an air cooled 
package, total power consumed in one package should be at 
most 1 Watt or a little more. Thus power dissipation per 
gate is limited to less than 0.2mW/gate for integration level 
of 5000 gates/chip, leaving some room for additional power 
dissipation in output buffers. An ED-MOS gate in which a 
depletion type MOS transistor is used as a load was selected 
as a unit cell for high speed, low supply voltage (5V or less) 
and high integration density. 

Based on preliminary experiments, delay-power product 
of a MOS gate with fan out of 3 was manipulated to be less 
than 0.4pJ for a LSI with a line width of 4^m or less. A 4/xm 
line width was assumed in 1972 to be one of the advanced 
MOS-LSI production standards of 1977. Thus the value of 
propagation delay time t pd per gate with small interconnec¬ 
tion stray capacitance and fan out of 3 was set to be less 
than 2ns. In a /upu LSI, the loading capacitance of each gate 
depends on logic design and pattern layout, particularly on 
wiring length, so I pd of a gate cannot be predetermined. The 
target value of t pd with fan out of 3 is one of the essential 
parameters to estimate device design and semiconductor 
process technology. Overall speed of a /xpu LSI largely 
depends also on its architecture and interconnections. 


Basic architectural design 

From a standpoint of architectural design, the 5000 gates 
integration level was one of the two biggest restricting fac¬ 
tors, because it was not considered large enough for our 
design aims of integrating a high-performance microproces¬ 
sor in one chip. 

Another big problem was the number of pins. If we em¬ 
ployed a commonly used package with less than 50 pins, we 
could not achieve a processing capability which would 
match the internal speed, because interface pins must be 
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used in a time-multiplexed fashion. Furthermore, instruction 
word-length is shortened and detailed control may be im¬ 
possible. Therefore, it was decided to adopt a flat package 
with 80 pins. The development of a high-performance inter¬ 
face to satisfy this condition, as well as the choice of func¬ 
tions to put in PULCE, were the most important design 
problems. 

In view of the technological level of semiconductor and 
computer systems, and the requirements imposed by the 
desired applications, we adopted the following considera¬ 
tions as the basis of the architectural design. 


Exclusion of sequence control 

As LSI fj^pu technology is based on mass production, its 
architecture must be general-purpose. Usually, processor 
functions (instructions) are divided into the following three 
categories, in which the relationships with general purpose 
requirements are different. 

(1) F type (Functional)—Arithmetic and logical process¬ 
ing functions. 

(2) P type (Procedural)—Sequence control functions. 

(3) E type (External)—External control functions. 

Requirements of the F type instructions do not differ in 
any applications as far as the microinstruction level is con¬ 
cerned. Accordingly, it is relatively easy to extract functions 
with wide generality for this type. However, each system 
entails a different requirement for the P type, especially with 
respect to interrupt processing, and hence P type standard¬ 
ization is difficult. Furthermore, because of the large number 
of interface lines and constraints on the amount of hardware, 
usually only simple functions, which may be unsatisfactory 
for the purpose of PULCE, can be integrated in an LSI 
chip. Finally, the E type has special features for each ap¬ 
plication. But, as these are external functions, only general- 
purpose considerations and interface expansibility need be 
dealt with for this type. 

In view of the above considerations, and because of con¬ 
straints on the level of integration and number of pins, we 
decided not to include a sequence control portion in the LSI 
chip. Instead we concentrated F type functions. Only the 
general-purpose requirements and flexibility of interfaces for 
external implementation of suitable P and E functions are 
considered. 


16 -bit basic word length 

The length of data processed at one time is very important 
for this type of /ipu. One method is to achieve expansibility 
by slicing a word into 4 or 8 bits, but there are items, like 
status information, which cannot always be handled well by 
the bit slicing, and even 8 bits may be too short for the 
integration level. Therefore, we have adopted 16 bits as 


basic word length and 32 bits for interface, and provided 
PULCE with some architectural features in relation to the 
simplification of double length data processing. 


Emulation oriented 

Here emulation means not only interpretation of machine 
instructions of existing computers, but also includes emu¬ 
lations of new system architectures or intermediate lan¬ 
guages directed to high level languages. This is one of the 
important features of new processors. For this purpose, 
mask registers and a few mode bits which are effective for 
extracting fields in a word are provided. Operating in con¬ 
junction with a multi-bit shifter, the mask registers serve not 
only for partial field processing as in (macro)instruction de¬ 
coding, but also for the usual processing. 


Stack function 

Pushdown stacks are now employed in various applica¬ 
tions and their effectiveness is widely recognized. There¬ 
fore, a special hardware support which always keeps the 
upper portion of a stack in internal registers is provided to 
increase its processing speed. 


Flexible control of hardware details by microprogram 

Generally speaking, as the level of a functional module 
becomes higher, it becomes more difficult to implement or 
modify a lower level function. In other words, modularity 
decreases flexibility. This is called “loss of transparency” 
due to modularization. 4 In the case of such high level mod¬ 
ules as PULCE, where general-purpose requirements are 
important, the loss of transparency must be kept as small as 
possible. For this purpose microprogram control by unu¬ 
sually long (32 bits) instructions has been adopted for flex¬ 
ible control of hardware details. 


Regular structure 

It is expected that this kind of jupu will be programmed 
by general users who are not always hardware specialists. 
This situation is different from the static microprogramming 
cases. Therefore, the organization of the instruction reper¬ 
toire and internal structure should be as regular and as easily 
understood as possible. 


ARCHITECTURE 

PULCE is organized in a 3-bus structure around the ALU 
and a shifter as shown in the block diagram of Figure 1. 
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Port 

Figure 1—PULCE block diagram (GPR: General purpose register, CTR: 

Counter, MDR: Mode register, STR: Status register, IFR: Interface 
register, FAR: File address register, FIR: File indirect register, FNR: File 
next register, INC: Incrementer, DEC: Decrementer, SEL: Bit selector, 
SW: Switch) 

Functional units 

Main registers 

The 16 registers listed in Table I are always directly ac¬ 
cessible and most of them are assigned dedicated functions 
as clearly shown by their names. However, FIR’s, FNR’s 
and IFR’s are provided with some special features described 
below. 

Each FIR is a word in a register file pointed to by FARO- 


TABLE I.—Main Registers 


Mnemonic 

Name 

Length(bits) 

Number 

hardware 3 

GPR 

General Purpose 

16 

4 

O 

FAR 

File Address 

4 

2 

o 

MDR 

MoDe 

*1 

16(4) 

1 

o 

*2 

STR 

StaTus 

16 

1 

o 

FIR 

File Indirect 

16 

2 

* 

IFR 

InterFace 

16 

2 

0 

FNR 

File Next 

16 

2 

X 

CTR 

CounTer 

16 

2 

o 


1, so no specific hardware exists for FIR’s. Each FNR is a 
word in a register file whose address is the content of cor¬ 
responding FAR plus one. Using both FIR and FNR, two 
continuous words in a register file can be processed without 
changing the FAR content. This feature is convenient for 
emulation of system architecture with word lengths of 32 or 
64 bits. On the other hand, when SM bit in MDR is set, 
FNRO is used for a stack function and its behavior differs 
from that described above (see Stack operation ). 

IFR’s are registers for communications with external 
hardware attached to PULCE and their input/output con¬ 
nections are made to interface ports via bidirectional buffers. 
The input/output through these ports are controlled exter¬ 
nally instead of by internal microinstructions. Furthermore, 
IFRO can be externally set/reset in a bitwise manner, and as 
a result, this register can be used as an external status 
register. In addition, if desired, a port of IFR1 can be directly 
connected to R-bus in PULCE by externally activating a 
special control terminal. This feature enables a user to either 
look at data on the R bus or put data on the R bus from 
interface ports. 

Register files 

Two register files (RFO and RF1) of 16 words are pro¬ 
vided. RFO is a general purpose register set, but it could be 
used for storage of the upper portion of the stack when 
MDR’s SM bit is set (see stack operation). The second 
register file RF1 is divided into two 8 words subfiles, RF1- 
0 and RF1-1. Seven words in RF1-1 can be used as mask 
registers. 

Arithmetic and Logic unit 

ALU consists of a binary parallel adder, 1-digit decimal 
adder/subtracter and logic circuits for AND, IOR, XOR, 
NOR and some special logic functions. 

Masking 

If designated by the MKfield of a microinstruction, data 
on L and R buses are ANDed with designated mask registers 
in RF1-1 before being loaded to ALU. ALU output may also 
be ANDed with a mask register before being put on the T 
bus. This feature facilitates effective extraction and pro¬ 
cessing of a partial field in a register. A trial coding of 
Church’s field processing problems 5 as shown in Figure 2 
proved the effectiveness of this feature. 

Shifters 

PULCE is provided with two types of shifters, a single 
word length multibit type and a one bit multiword (up to 4 
words) shifter. The former can shift a single word 1-15 bits 
in one machine cycle. Cooperating with mask registers, it 
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Bits 

Location A 
A+2 
A+4 


0 0 '5 
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a(7) b (9) 
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e(15) f 

g(l6) 
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,1(8) k(8) 
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Problem 2: Coding Results 
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(b) Problem 2 


Figure 2—Coding results of partial field reformatting 
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efficiently extracts partial field in a word. General purpose 
registers GPRO-3 are equipped with a one bit long shifter, 
and contents of the selected one to four GPR’s can be shifted 
as a one long word. It is effective for multiplication and 
division. 

Stack operation 

Since constraints on volume of hardware rule out the use 
of an independent stack, provision is made for a special 
hardware support (called Ring Stack) which keeps an upper 
portion of the stack in register file 0. Thus, when the MDR’s 
SM bit is set, RFO is accessed by FNRO in the following 
manner. 

(1) When FNRO is assigned in the T field of an instruction, 
FARO is incremented at the beginning of its execution. 
Then at the end of its execution MDR’s SE bit is reset, 
and if FARO equals 15 (when MDR’s SB is reset) or 
FARO equals 7 (when SB is set) at this time, STR’s F 
bit which indicates “stack full” is set. 

(2) When FNRO is assigned in the R field, FARO is dec¬ 
remented at the end of execution, provided that SE 
is reset at the beginning. Furthermore, if FARO equals 
15 (when MDR’s SB is rest)/7 (when SB is set), STR’s 


E bit which indicates “stack empty” is also set. How¬ 
ever, if SE is set at the beginning, E bit is set and this 
instruction is ignored. 

As is easily seen, (1) and (2) of the above operation cor¬ 
respond to “push” and “pop” operations of the stack. SE 
is a bit which is set by a microprogram when the stack is 
completely empty, and by using this bit, stack underflow is 
detected. SB is used to alternate the set conditions of STR’s 
E and F bits between the middle and the bottom of RFO, 
essentially making it a ring of registers (see Figure 3a). 
Accordingly, by saving 8 words in RFO in external memory 
when F is set and restoring these 8 words to RFO when E 
is set, an upper portion of the stack can always be kept in 
RFO (see Figure 3b). 

Simulation of Ring Stack has been done and the results 
are shown in Figure 4. The probability of data transfer to 
and from external memory is substantially reduced by the 
Ring Stack, when the number of continuous push/pop op¬ 
erations is less than the hardware stack size (16). 

Microinstructions 

The microinstruction repertoire is designed not only for 
ease of application but also for controllability of the /xpu. 
There are six formats shown in Figure 5. 


Stack 



b) Interrupt processing flow. 

Figure 3—Ring stack mechanism 
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Average number of continuous "push" or "pop". 

Figure 4—Performance evaluation of the ring stack 


RR type instructions 

This type is used for operations between registers or a 
register and 5-bit literal. 

Operation code (RR)—RR designates any one of 16 opera¬ 
tions consisting of 4 logical operations, 6 binary addition/ 
subtraction, 4 decimal addition/subtraction and two special 
operations for multiplication and division. 

Status assignment (STB, STC)—STB assigns a bit in STR 
and the next instruction may be ineffective depending on its 
value. STC bit complements skip condition. 

Status save (SS)—SS bit indicates whether status informa¬ 
tion after execution is saved in STR or not. This control 
enables the programmer to retain past status information in 
STR. 

Bus control (TC, LC, RC)—TC controls loading of data on 
the T bus into an assigned register, and RC designates 
whether the R field is a literal or register number, while LC 
indicates whether the assigned register content or “0” is put 
on the L bus. 

Counter control (CT)—CT is a field to indicate that one or 
both of two counters are decremented. It is mainly used for 
loop control, 

RE type instructions 

These instructions are for operations between a register 
and a 16-bit literal. The lower 16 bits (E) of the instruction 
are interpreted as a literal and put on the R bus. In addition 
to normal arithmetic and logical operations, RE type is pro¬ 
vided with a special operation called pattern match which 
puts on the T bus an AND of 16 bits information decoded 


from the lower 4 bits of L input and a literal (E). This is 
useful for classifying codes, detecting signs in decimal op¬ 
erations, etc. 

RH type instructions 

These are operations between a register and an 8-bit lit¬ 
eral. As this type can take advantage of the masking feature, 
it is useful for processing of 8-bit data. 

SH type instructions 

Instructions of this type shift the content of a register 
assigned in the T field by the amount (maximum of 15 bits) 
indicated by the R input in one machine cycle. In addition 
to normal shift modes, there is one special logical shift in 
which GPRO is implied as a source register. This is often 
useful for processing of partial fields in a register, because 
the original content of the register is not changed by the 
instruction execution. 

LS type instruction 

This is a long shift instruction of GPRO-3. Assigning a 
repeat mode in the MM field, only this instruction can be 
repeated until the designated counter becomes zero or the 
skip condition is fulfilled. 

NP type instructions 

Instructions whose upper 3 bits equal “111” do not have 
any effect internally. In other words, NP type is designed 
individually for each system to control sequence and exter¬ 
nal operations. 

SEMICONDUCTOR TECHNOLOGY 
n MOS/SOS technology 

To actually produce PULCE from the design we have 
described, devices, that are of high integration density, of 
low power consumption, of high speed and of small stray 
capacity had to be developed. Considering the state of the 
art in semiconductor technology, n -channel E/D MOS cir¬ 
cuits on sapphire substrate (n MOS/SOS technology) seemed 
to be the most suitable device for this LSI. Especially small 
stray capacitance associated with n + diffusion layer wiring 
and thin film wiring on an insulating sapphire substrate is 
suitable for a /upu in which long parallel bus lines meander 
on the substrate. 

Most of the studies of silicon on sapphire technology have 
been concentrated on CMOS/SOS. 6 There has not been any 
attempt to fabricate high density n- channel MOS logic cir¬ 
cuits utilizing SOS technology, but an n -channel E/D gate 
is considered to be superior to a CMOS gate because of its 
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Figure 5—Microinstruction formats (RR: Op. code, T(L): T&L bus, ST: 
Status, MK: Mask control, SS: Status save, TC: T bus control, RC: R bus 
control, LC: L bus control, CT: Counter control, R: R bus, RE: Op. code, 

high packing density and design simplicity. So, before the 
final fabrication of the PULCE LSI, three years of R&D 
were spent developing the high density n -channel MOS/SOS 
process. The method of polishing the sapphire substrate, the 
hetero-epitaxial thin silicon film growth condition for re¬ 
duced defect density, the research on origins of drain leak¬ 
age current and the stabilizing of fine line photoengraving 
conditions on SOS wafers were included in the R&D project. 

Preliminary experiments 

Before actual fabrication of the PULCE LSI, functional 
blocks such as register and arithmetic logical unit RALU 
and register file RFO were developed as experimental de¬ 
vices on SOS. The RALU which contains about 2400 gates 
consists of 16 words by 16 bits local storage, 16 bits adder, 
two working registers and control circuits. The chip size is 


0 


E: Emit, RH: Op. code, HL: High-low control, SHT: Shift type, SHD: 
Shift direction, SI: Shift-in control, MM: Multimode, RS: Register select) 


5.00x4.72mm using 6 pm design rule. 7 The typical propa¬ 
gation delay time through 11 NOR gates in carry circuits is 
70ns, one third of that on bulk silicon. Register file RFO 
consists of 16 words by 16 bits register file, 4 bits file address 
register FARO, input buffer circuits and three (T,R,L) bus¬ 
ses. This register file performs complex functions such as 
writing data on the T bus and reading out data on the R and 
L bus simultaneously. The design rule used is the same 4 pm 
rule as that of the final design. The chip size is 4.41 x4.74mm 
and 1,300 gates are included. The access time is about 30ns 
or less as shown in Figure 6 and is within 20ns of the design 
target of 50ns. 

The PULCE LSI is fabricated by 4pm design rule and 
coplanar process. Figure 7 shows the relation of average 
propagation delay time f pd and power dissipation P A of the 
n MOS/SOS gate. The parameters of the figure are the ratio 
of the channel width to the channel length of load transistor 
of the inverter ( 4pm : 4pm and 4pm: 3pm), and the thickness 
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__V T0 (VOlt) ; 

Figure 6—The access time of register file RFO 


of gate oxide, 750A and 770A respectively. This figure shows 
the difference of the f pd P d product on SOS (0.21pJ) and on 
bulk silicon (0.34pJ) in the same design pattern. It is clear 
that SOS is faster than bulk silicon by a factor of 1.6 at the 
same power dissipation. 


Fabrication of the PULCE LSI 

For the fabrication of PULCE LSI new technology such 
as the coplanar process on SOS which improves low gate 
breakdown voltage and prevents extra drain current in or¬ 
dinary SOS structure, 8 and circuits with four types of tran¬ 
sistors as shown in Table II which reduce power dissipation 
were used. The gate used E/D circuits, the threshold voltage 
of which is 0.8±0.2V (V T e) for E type transistors and 
-2.0±0.3V (V TD ) for D type transistors. In addition to these 
transistors, two other threshold voltage transistors were 
used. The first one was the load transistor of a memory cell 
which was fabricated by boron and phosphor ion implanta¬ 
tion. The threshold voltage V TED is -0.5V. Power dissipation 
of memory cells in the register file was reduced by a factor 
of ten compared to all the same D type transistors. The 
another threshold voltage transistor was the intrinsic tran¬ 
sistor (I type transistor). The pushpull driver using a D type 
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Figure 7—The relation of average propagation delay time vs. power 
dissipation for n MOS/SOS gates 


load transistor to drive the large capacitance of bus lines 
requires power consumption at low output voltage level. 
Using an E type load transistor causes the voltage to drop 
to the level of the threshold voltage. To solve this problem, 
an I type load transistor for which no ion implantation tech¬ 
nique was used was incorporated in the pushpull line driver. 
It has a threshold voltage V TI equal to —0.5—1-0.3V. Power 
dissipation was reduced by a factor of ten and by using a 
large size load transistor the circuit speed was kept at the 
same level. 


TABLE II. —Four Types of Transistors 


Type 

Ion implantation 

Typical V T 

Boron 

Phosphorous 

E 

O 

X 

0.8V 

D 

X 

O 

-2.0V 

I 

O 

X 

-0.2V 

E-D 

X 

O 

-0.5V 
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Figure 8—Photomicrograph on the PULCE LSI 

Figure 8 shows the photomicrograph of a PULCE LSI. 
The chip size is 8.85x6.66mm. There are 7,000 logic gates 
and 20,000 transistors. The processor has a 200ns microin¬ 
struction cycle time using a single phase clock. The delay 
through the look ahead carry path of 10 gates for a 16 bits 
arithmetic operation is 35ns and the binary addition time 
including decoder delay ranges from 80ns to 115ns. Multibit 
shifter delay is 30ns. The chip uses a single 5V power supply 
and dissipates 1.5W at its full rated speed. Figure 9 is a 
photograph of the 80 pins package which has five aluminum 
cooling fins. The heat resistance is 10° C/W at 1 m/sec air 
flow. 

CONCLUSION 

We have described the architecture of a new high-perform¬ 
ance microprocessor PULCE and the semiconductor tech¬ 
nology developed for its implementation. The first function¬ 
ally complete LSI PULCE was constructed in the fall of 
1977 and is now being evaluated. The performance of LSI 
PULCE is summarized in Table III. 



Figure 9—Photograph of 80 pins package 


TABLE III.—Summary of the Performance of PULCE 


Device type 

n MOS/SOS 

Chip size 

8.85 x 6.66mm 

Gates in a chip 

7000 

Transistors in a chip 

20000 

Package 

80-pin flat package 
with cooling fins. 

Power supply 

5V 

Machine cycle 

200ns 

Power dissipation 

1.5W 

Operating temperature 

0°C-50 t> C 

Data width 

16bits 

Microinstruction 

32bits 

supplied from outside 

Registers 

(General purpose) 

(Mask) 

(Dedicated) 

44 

29 

7 

(16bits) 6 

(4bits) 2 

Shifter 

(Single word) 

(2,3,4 words) 

0-15bits 

lbit 

Decimal operation 

add/sub (1 digit) 

Stack 

Hardware support 

Multiply/divide 

No hardware as such, but 
special hardware available 
to support these instruction 


Before the actual implementation of LSI PULCE, we built 
several experimental machines with PULCE architecture by 
using TTL SSI’s and MSI’s, and they have been successfully 
used to construct processor modules of experimental mod¬ 
ular multiprocessor systems. 2,9 

Now we are planning to use LSI PULCE in many research 
areas, including a data base machine, a LISP machine, a 
character recognition system and a polyprocessor system. 
Among these the first application will be a high-performance 
personal computer, which will be completed by the first 
quarter of 1978. 
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CACS—Urban traffic control system 
featuring computer control 


by TORU MIKAMI 

Nippon Electric Co., Ltd., 
Kawasaki City, Japan 


INTRODUCTION 

In Japan, the number of automobiles has exceeded thirty 
million and the number of driver’s license holders reached 
thirty-five million (approximately thirty-five percent of the 
total population) at the end of 1976. Automobile traffic in 
many cities has been increasing rapidly year by year. Au¬ 
tomobiles and trucks, at present in Japan, perform the most 
important part of a means for transporting passengers and 
goods. In fact, for seven years, the amount of transportation 
by automobiles exceeded that by railways in Japan. 

Increased use of automobiles in cities, on the other hand, 
has caused many serious problems: traffic congestion, traffic 
accidents, air pollution from exhaust gas etc. For example, 
more than thirty persons are killed by automobile accidents 
every day. These problems impose great losses and strains 
on city dwellers. The city traffic problem has become one 
of the most important social problems in Japan. 

To solve this traffic problem, many means and systems 
have been considered and applied to the cities. Area traffic 
signal control systems have been installed in the major cities 
of Japan. Many big cities have also been equipped with air 
pollution surveillance systems. Enforcing traffic regulations, 
extending safety facilities, developing new city transporta¬ 
tion systems and many other means have been tried. 

It cannot be said, however, that the situation has changed 
for the better. Both time involved and area covered by traffic 
congestion in the city are still increasing. As a result even 
with an increase in the number of vehicles, the transporting 
capacity of automobiles in the city is decreasing. 

In order to solve the city traffic problem, it is basically 
necessary, in addition to the above, to consider a means for 
increasing efficient utility of the road network in a city. For 
this purpose, Comprehensive Automobile Control System 
(CACS) project, which is introduced in this paper, was es¬ 
tablished and is continuing. The purpose of the CACS proj¬ 
ect is to develop an effective means and system for con¬ 
trolling the city automobile traffic comprehensively and 
efficiently. Technically, this project aims to develop a total 
information processing system for the automobile traffic by 
combining car electronic device, car-ground communica¬ 
tion, computer control and network technologies. This sys¬ 


tem will be used in. coordination with the existing traffic 
control system mentioned above. 

The CACS project is financed by the Agency of Industrial 
Science and Technology, Ministry of International Trade 
and Industry. The project started in 1973 and is planned to 
continue six years. More than twenty million dollars have 
been invested up to now. Organizations involved in this 
project are the Research Association for Comprehensive 
Automobile Control Technology, Toyota Motor Co., Ltd., 
Nippon Denso Co., Ltd., Sumitomo Electric Industries, 
Ltd., Nippon Electric Co., Ltd., Hitachi, Ltd. and others. 


SYSTEM OBJECTIVES AND FUNCTIONS 

In the CACS project, for analysis of city automobile traffic 
problems, the following four system objectives have been 
adopted: 

1. Relieve vehicle traffic congestion. 

2. Reduce exhaust pollution that accompanies traffic 
congestion 

3. Prevent automobile accidents 

4. Increase public and social applicability of automobile 
use. 

In order to achieve these objectives, the following four 
functions have been set up as technology development target 
functions: 

1. Provide optimal route guidance to moving vehicles 
while preventing congestion and resultant exhaust pol¬ 
lution during the traffic congestion period 

2. Provide travel priority to emergency and public service 
vehicles 

3. Provide drivers with driving information pertinent to 
safe driving 

4. Provide drivers with other advance information on, for 
example, the state of traffic emergency in a specific 
area. 

In the actual CACS project development phase, these 
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target functions are realized through the development of five 
subsystems shown below: 

1. Route guidance subsystem—A system which guides 
drivers to their destinations via optimal routes accord¬ 
ing to traffic conditions by displaying guidance infor¬ 
mation visually inside vehicles. 

2. Route display board subsystem—A system which gives 
drivers the information on road congestion status and 
recommends routes by outside visual displays. 

3. Driving information subsystem—A system which 
brings to drivers atb ruion information on traffic regu¬ 
lations and driving warnings by inside indications. 

4. Traffic incident information subsystem—A system 
which transmits accident information and the like to 
drivers for the area through which they are driving by 
car radios enforced to receive concerned signals. 

5. Public service vehicle priority subsystem—A system 
which allows public service vehicles, such as ambu¬ 
lances, fire engines and repair trucks pass through with 
priority. This system is to operate in conjunction with 
signal control systems. 

These five subsystems constitute Comprehensive Auto¬ 
mobile Control System. Figure 1 shows a conceptual con¬ 
figuration of the CAC system. Figure 1 also shows potential 
relationships in the future between CAC system and repre¬ 
sentative external systems, such as area traffic signal con¬ 
trol system, expressway surveillance system and others. 
Traffic administration and road administration also have im¬ 
portant relations to the CAC system operation. 



-■» Flow of control, I not ruction and command 

-Flow of Information 

Figure 1—Comprehensive automobile control system concept 



PILOT SYSTEM TEST 

A pilot system test was conducted in the CACS project 
for a year beginning in October of 1977. Purposes of this 
pilot system test are to: 

1. Evaluate the effectiveness of developed technologies 
in city streets 

2. Collect traffic data for future system design 

3. Evaluate social acceptability of this system. 

The pilot test area is a 28-square kilometer area in the 
southwestern section of Tokyo (see Figure 2). This pilot 
area contains a road network representative of the road 
network of the entire city; 89 main intersections in this area 
(including 15 expressway ramps) have been chosen as guid¬ 
ance intersections for route guidance. Three hundred thirty 
test vehicles, which are equipped with full sets of vehicle 
units, and 1,000 data collecting vehicles, which are equipped 
with transmitting parts of vehicle units only, are planned to 
be used in the pilot test. 

PILOT SYSTEM FEATURES 

The pilot system is composed of the following five sub¬ 
systems: 

• Route guidance subsystem (RGS) 

• Driving information subsystem (DIS) 

• Traffic incident information subsystem (TIS) 

• Route display board subsystem (RDB) 

• Public service vehicle priority subsystem (PYP) 
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Figure 8 shows the system configuration which realizes 
functions of these five subsystems. 

Main features of the pilot system configuration are as 
follow: 

1. The route guidance subsystem forms the central part 
of the pilot system. The realtime data on traffic con¬ 
ditions are collected by the route guidance subsystem 
and are used by other subsystems. The guidance indi¬ 
cation is fed to test vehicles only in the pilot area but 
any point in 23 wards of Tokyo can be designated as 
the destination. 

2. The driving information subsystem mostly shares the 
same equipment with the route guidance subsystem. 

3. The traffic incident information subsystem and route 
display board subsystem also utilize the equipment, 
especially computers, of the route guidance subsystem. 

4. For the public service vehicle priority subsystem, only 
the function of identifying public service vehicles and 
their locations is realized. 


ROUTE GUIDANCE SUBSYSTEM 
Destination code 

A driver who wants route guidance subsystem service 
determines a destination code number corresponding to his 
desired destination by looking at the guidance road network 
map of the pilot system. The destination code is represented 
by a seven digit number. The first two digits represent a 
region whose area is approximately equal to that of prefec¬ 
ture (2,OOOkm 2 — 14,000km 2 ). A section (whose area is, as in 
the case of Tokyo, about 30km 2 ) in the region is represented 
by the first four digits. Each section is divided into less than 
seven zones (the fifth digit) and the zone includes less than 
sixty-three points (destinations) (the last two digits). Each 
point (destination) corresponds to individual sides of the 
road between intersections. 

Vehicle unit and roadside unit 

Each test vehicle is equipped with a vehicle unit. The 
vehicle unit consists of a destination encoder, display unit, 
vehicle loop antenna and control unit (see Figure 3). The 
driver, at the start of his trip, sets the recognized destination 
code number on the destination encoder. The driver can 
also select the use of an expressway by pushing an express¬ 
way option button. When the vehicle passes over a road 
loop antenna buried near a roadside unit before entering a 
major intersection, the destination code number and ex¬ 
pressway option information are transmitted to the roadside 
unit through the vehicle antenna, along with other infor¬ 
mation, such as vehicle unit number and vehicle class code. 
A roadside unit is installed near each one of major intersec¬ 
tions in the pilot test area. The roadside unit consists of a 
roadside control unit and a set of road loop antennas laid 
under the road in each arm of the intersection. Each roadside 



Figure 3—Vehicle unit and roadside unit 


unit is connected to a computer center with a communication 
line (see Figures 3 and 5). 

Guidance indication 

The information from the vehicle unit is fed into the road¬ 
side control unit with the information identifying the road 
from which the vehicle is entering into the intersection. 
Using the destination code number, expressway option in¬ 
formation, vehicle class code and entering link number as 
key words, the roadside control unit searches a guide table 
stored in the uhit and keys guidance information correspond¬ 
ing to the request. The guidance information is sent back to 
the vehicle unit through antennas and is indicated on the 
display unit attached to the dashboard. The indicated infor¬ 
mation includes the following (see Figure 4): 

a. Intersection shape (intersection indicator emblem in 
Figure 4) 

b. Turning direction (indicated with one of seven arrows; 
the case in Figure 4 indicates a turn to the right) 

c. Lane selection before entering intersection 

d. Lane selection after passing intersection 

e. Slope (UP: uphill, DOWN: downhill) 

f. Entrance to expressway 

g. Exit from expressway 

h. Emergency guidance (detouring instruction) 

i. Destination arrival (indicated by flashing arrow). 
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Driving in accordance with the guidance indication of this 
subsystem from the start point to the destination enables the 
driver to reach the destination in the minimum amount of 
time even though he is unfamiliar with the place. This guid¬ 
ance indication is automatically updated every fifteen min¬ 
utes according to changes in present and predicted future 
traffic conditions. 


System configuration 

Figure 5 illustrates the route guidance subsystem config¬ 
uration. One hundred and three roadside units installed at 
major intersections in the pilot test area are connected to a 
communication control computer in a control center with 
1200 bps speed data communication lines. Six computers 
are installed in the control center: a communication control 
computer, central computer, operation control computer, 
traffic network controller and two route search computers. 
A traffic network simulater performs a substitute for the 
route search computers. One hundred and eighty two con¬ 
ventional traffic flow detectors are connected to the system 
in order to measure the vehicle flow rate and road occu¬ 
pancy. 

System operation cycle 

General guidance information fed to drivers is automati¬ 
cally updated every fifteen minutes. Every processing re¬ 
quired for its renewal, such as traffic data processing, trav- 



*182 

Figure 5—Route guidance subsystem configuration 


eling time estimation (arc costs), optimal route computation, 
guide table formation and so on is consequently carried out 
in fifteen minute cycles. Some other processings, for ex¬ 
ample diagnoses of roadside unit efficiency, are also exe¬ 
cuted every fifteen minutes. The data on traffic flow and 
road occupancy are collected from detectors at five minute 
intervals. The information from vehicle units is gathered at 
one minute intervals. Lengths of these cycles can be 
changed by commands from an operating console. 

The basic operation of this subsystem consists of a se¬ 
quence of the following three cycle operations. The data on 
traffic flow, density and running test vehicles are collected 
in the first fifteen minute cycle. Upon these data, the optimal 
paths are computed and the guide tables are made in the 
second fifteen minute cycle. In the third cycle, the guidance 
for the vehicle is performed according to these guide tables. 
Therefore, the vehicles are eventually guided on the basis 
of data for two cycles before. 


Information flow and processing 

Figure 6 shows an outline of the information flow and 
processing in this subsystem. The communication control 
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Figure 6—Information flow and processing in route guidance subsystem 
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computer collects data on points and times of communica¬ 
tions between roadside units and equipped vehicles from 
roadside units and sends the data to the central computer. 
The data on traffic flow rates and traffic densities, detected 
by the detectors, are also collected from the roadside units 
and a traffic data processing unit and sent to the central 
computer. 

In the central computer, an arc cost for each segment of 
the road network is estimated and predicted on these col¬ 
lected data. Here, the “arc cost” represents the length of 
time required to traverse the distance from one communi¬ 
cation point to the next communication point beyond the 
intersection. These arc cost data are sent to the route search 
computers or to the traffic network simulator. 

The route search computers compute optimal paths using 
the arc cost data, pertinent area traffic regulation data and 
other network data. The computation of optimal route 
search is performed at each intersection. Time-minimal 
paths from each intersection (origin) to every point (desti¬ 
nation) in the network are computed. For computation econ¬ 
omy, the route search network is specially deformed. The 
network structure is expressed most precisely near the start¬ 
ing point (origin) and its expression becomes less precise as 
the field goes away from the starting point. 

Computation results are condensed into a guide table from 
every guidance intersection. The guide table is expressed in 
a combination of list and table structures. The maximum 
size of the table is less than two kilowords (one word is 
sixteen bits). The guide table thus formed is transmitted to 
each roadside unit through the 1200 bps speed data com¬ 
munication line. The use of the guide table at the roadside 
unit was previously mentioned. 

OTHER SUBSYSTEMS 

Driving information subsystem 

The driving information subsystem gives the driver infor¬ 
mation helpful to safe driving by inside display and audible 
signal. When a vehicle equipped with the vehicle unit passes 
by a buried loop antenna connected to a roadside unit, 
pertinent area traffic regulations and other information on 
the road the vehicle is traversing is transmitted to the vehicle 
unit and any pertinent warning information is shown on the 
display unit. The warning information is expressed in one of 
the following messages (see Figure 4): 

1. Pedestrian crossing ((T) in Figure 4) 

2. Go slow 

3. Change in lane condition 

4. Stop 

5. Change in width of road 

6. Road under construction. 

The information on the posted speed transmitted from the 
roadside unit is stored in the vehicle unit, which checks the 
vehicle’s cruising speed. When the vehicle speed exceeds 
posted speed limit, an audible warning signal is emitted. 


The contents of this driving information, stored in the 
roadside unit, is updated by the master information in the 
central file at every system cycle. The contents can also be 
changed by intervention command from the control room. 

Traffic incident information subsystem 

The traffic incident information subsystem informs a 
driver about information on emergencies which affect traffic 
flow by means of area-limited broadcasting. The broadcast 
information includes the news on traffic accidents, fire, sud¬ 
den traffic jam and the like. It also contains advice for the 
driver such as recommended route, foreseen trip time etc. 

In an emergency, the above information, a part of which 
is made up of arc cost data from the route guidance subsys¬ 
tem, is immediately transmitted from the control center to 
pertinent TIS roadside units. Each TIS roadside unit broad¬ 
casts the information to vehicles through the leaky coaxial 
cable antennas (LCX). When a vehicle having a car radio 
equipped with TIS radio adapter enters into LCX range, the 
vehicle receives the announcements. These announcements 
interrupt regular radio broadcasts and are received even 
though the radio is switched off. 

Route display board subsystem 

The route display board subsystem furnishes data con¬ 
cerning traffic conditions to all drivers of vehicles which are 



Figure 7—Route display board 
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not equipped with vehicle units. The information includes 
data on traffic congestion, accidents and/or recommended 
routes. This data is indicated on route display boards in¬ 
stalled along the road. The names of a number of neighboring 
intersections and arrows connecting them are illustrated on 
the board (see Figure 7). The traffic conditions on the route 
between two intersections connected by an arrow are indi¬ 
cated by the colors of the arrow: 

1. Red: heavy congestion 

2. Yellow: slight congestion 

3. White: no information 

4. Green: recommended route to a major intersection 

The congestion information and recommended routes re¬ 
sult from arc cost data calculated in the route guidance 
subsystem. They are sent from the central computer system 
to the route display boards through off-line operator han¬ 
dling. 

Public service vehicle priority subsystem 

The function of this subsystem is not realized in the pres¬ 
ent pilot system. Only the capabilities of detecting and track¬ 
ing the equipped public service vehicles on the central wall 
type display are checked. 


CONTROL ROOM 

Figure 8 is a sketch showing total CACS pilot system 
configuration. A control center is located nearly at the center 
of the pilot test area. One large-scale computer (NEAC 2200/ 
375), five minicomputers (four NEAC 3200/70 and a HIDIC 
350), one special-purpose computer and four specific use 
processors are installed in the control center. 

A control room is included in the control center in order 
to control and manage operation of the overall system and 
to supervise pilot test execution. The major devices installed 
in the control room are a supervisory operating console, 
four subsystem operating consoles and a wall display (see 
Figure 9). The wall display includes a central screen display 
(2.0mxl.5m) with a projector in the rear, color character 
CRT display, lamp panel and a few indicators. 

With the aid of these devices, the following operations are 
performed in the control room: 

1. Initial system set up 

2. Starting and stopping the overall operation or partial 
function of the system 

3. System function intervention after an accident, in an 
emergency situation etc. 

4. Recording system operating data 

5. Roadside unit status diagnosis 

6. Changing system parameters, configuration etc. 


CONTROL CENTER 



(oetector) (detector ) 


Figure 8—Pilot system configuration 
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Figure 9—Control room 


7. Monitoring traffic conditions 

8. Monitoring system and device status 

The congestion degree on each road in the pilot test area, 
public service vehicle location, optimal route and like data 
are indicated by colors on the map projected on the central 
screen of the wall display. Information pertinent to tracking 
a designated vehicle can also be illustrated there. Roadside 
unit and detectors status data are indicated on the character 
CRT display. In addition, the computer system’s status, test 
conditions, data and time are shown on the wall display. 
Most of these figures can also be seen on the color graphic 
display mounted on the operating console. 


CONCLUSION 

The Comprehensive Automobile Control System has been 
described. Essentially, the description has concentrated on 
its pilot system. 

It is expected that the CAC System will produce many 
beneficial effects on individual drivers and society to: 

1. Lighten the driving effort of a driver who is unfamiliar 
with an area by guiding him on the optimal route to his 
destination 

2. Reduce the time required for traveling from the start 
point to a specific destination 

3. Relieve local traffic congestion in a city 

4. Decrease traffic accidents due to careless driving 

5. Eliminate the gasoline waste caused by engine idling 
in traffic congestion 

6. Reduce air pollution due to exhaust gas. 
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A distributed processing system and its application to 
industrial control 
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INTRODUCTION 

Industrial computer systems are characterized with some of 
their native features major items of which are: 

1. Large amounts of information come from physical 
equipment, not from human objects. 

2. The systems must control physical equipment having 
no human-like intelligence. 

3. Quick response or parallel response is a great concern 
in most cases. 

4. Reliability of computer systems and safety considera¬ 
tion to operators and equipment in case of computer 
malfunction are vital elements. 

More than 20 years have passed since the first installation 
of computers in process industries. Several steam power 
plants and some of steel industries were the volunteers in 
Japan. Table I summarizes the features characterizing the 
progress of the process control system. 

There were some significant milestones during the prog¬ 
ress of which the remarkables were the appearances of min¬ 
icomputers, micro-computers and data high-way. 

Technical innovations made after each of these milestones 
have pushed designers toward the preference for distributed 
configuration of both system architecture and software. 

The attempt to actualize various types of dual, duplex, 
hierarchical systems using multiple numbers of computers 
connected through channel controllers or memory bus 
adapters was motivated by the advent of mini-computers 
beginning in 1970. 

Data high-way is brought into real world as the most 
useful communication tool to connect computers in medium 
distance apart. It had the reason of its birth in the process 
designers’ demands that more reliable coupling means with¬ 
out losing economical properties than the one they had in 
using communication lines were desired. CAM AC data way 
and IEC bus are also used in subsystems to connect instru¬ 
ments and some kinds of actuating devices. 

Micro-computers have been welcomed by process design¬ 
ers in quick rise. The word “distributed control system” 
used to characterize micro-computer-controlled digital con¬ 
trol systems has become popular since 1975. This system 


has micro-computers in their hearts instead of conventional 
logic elements. 

The functional requirements to the computer systems be¬ 
fore 1975 were entirely purposed to control physical objects 
such as valves and motors. However, since 1976 increasing 
amounts of jobs to handle those kinds of information con¬ 
cerned with management control and production control 
have come within the scope of process control system. For¬ 
tunately the computers called mega-mini or some others 
which have main memories sizing more than 256 KB have 
been brought into market. These have been accepted as the 
suitable intervening computers between conventional phys¬ 
ical control systems and the management information sys¬ 
tem. In this paper, these kinds of computers are called 
information control computers. 

The latest design on the distributed computer system cov¬ 
ering the scope of information control through to physical 
control and some typical applications to some real industries 
are described in this paper. 

SYSTEM CONFIGURATION 

Figure 1 shows fundamental configuration of the distrib¬ 
uted processing. Computers (CPU), remote scanners (PI/O) 
and devices are connected to a data high-way through re¬ 
mote stations. The data high-way adopted here (TOSHIBA 
TOSWAY 1500 or 6300) is loop type. This type of the data 
high-way has the following characteristics: 

1. Dual lines are applied to have good fault-tolerancy 
against line failures. 

2. M:N communications are available. 

3. Twisted paired cable or telephone line may be used. 
PCM type communication is adopted. 

4. Protocol is based on ISO-HDLC. Messages are trans¬ 
parent. 

5. Cyclic error checks are made. 

6. Speed is 6.3 Mbps for high speed type (TOSWAY 6300) 
and 1.544 Mbps for low speed type (TOSWAY 1500). 

7. Maximum numbers of remote stations are 32 for a loop. 
Maximum numbers of devices for a station are 16. 

8. Maximum length permissive between adjoining remote 
stations is 4 Km. Maximum total length is 100 Km. 
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TABLE I.—History of Computer Control Systems 


AGE 

1958 

COMPONENT 

LOGGER 

LOGIC 

ELEMENT 

GERMANIUM 

TRANSISTOR 

SYSTEM 

FUNCTION 

DATA 

LOGGING 

TOPICS IN 

SOFTWARE 

TECHNIQUE 

MACHINE 

LANGUAGE 



TYPI CAL 
SYSTEM 
CONFIGURATION 


COMPUTING PROCESS 
LOGGER COMPUTER 


SILICON 

TRANSISTOR 


OPERATOR SET UP 
GUIDE CONTROL 


ASSEMBLER MULTI DDC 

PROGRAMMING AIGORITHM 

PROCESS 

MODEL 


MINI 

COMPUTER 






D-PROCESSOR, MEGA MINI, 
CROSS CALL NETWORK 
DEVICE ARCHITECTURE 


GSI 



PROCESS 

FORTRAN 

PACKAGED 

SOFTWARE 


PRODUCTION TOTAL DISTRIBUTED 

CONTROL AUTOMATION CONTROL 


FILE SYSTEM op nl mpn HIGHER LEVEL 
SP.DL.meo LANGUAGE. 



NOTES;SCC(SUPERVISORY CONTROL),DDC(DIRECT DIGITAL CONTROL).PC(PROCESS COMPUTER),BC(BUSINESS COMPUTER) 


Computers are connected with data high-way through 
DMA adapter and controllers. The heart of each remote 
station is made of micro-processor with 16 bits configura¬ 
tion. 

Micro-processors are also used in the remote scanners to 
make them flexible to requirements variety and intelligence. 
Another noteworthy application of micro-processors is the 



Figure 1—Fundamental system configuration 


one to micro-controllers. The purposes that micro-control¬ 
lers have are PID (proportional-integral-differential), direct 
digital control and sequential control of equipment such as 
motors, valves and actuators. These are used to control 
positions, speeds and sequential timings. A 16 bit micro¬ 
processor (TOSHIBA 40L) is adopted to the micro-control¬ 
ler. 

Figure 2 shows an extended system in which we have one 
master station (MST) and four remote stations (RST). A 
mega-mini computer with functions to control management 
information is connected to the master controller. This in¬ 
formation control computer is connected also to business 
computers (BC) of upper level through communication lines. 
Four micro-controllers, with the functions of PLC (pro¬ 
grammable logic control), RC (remote process I/O control) 
and PI/O (process input and output), are connected to re¬ 
mote stations. Remote process input and output (RPI/O) 
devices or remote scanners are controlled by micro-con¬ 
trollers. 

The information control computer or mega-mini (TO¬ 
SHIBA series A70) is a 32 bit computer with 1 Mbytes 
(addressable max. size is 16 Mbytes) main memory. This 
computer not only provides advantages that will be obtain¬ 
able by minicomputer architecture, but also provides capa¬ 
bility to process large numbers of data with high precision. 
This is really suitable to exist in the medium level between 
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MANAGEMENT 

INFORMATION PRODUCTION INFORMATION 

CONTROL CONTROL SYSTEM 



PROCESS PROCESS PROCESS PROCESS 


PLC ;PROGRAMMABLE LOGIC CONTROLLER 
RC ;REMOTE PI/O CONTROLLER 
RPI/O I REMOTE SCANNER 

Figure 2—An example of distributed computer system 


process control system and production control or manage¬ 
ment control system. 


SOFTWARE 

An individual operating system resides in each component 
computer of a distributed system and each operating system 
works independently. Application programs to be applied to 
all the component computers are described with a unique 
high-level language designed for process control. The lan¬ 
guage (TPL) is a subset of PL/I plus process control func¬ 
tions. 

Specific features are: 

• Addition of case statement 

• Labeled variable and multiple entrance and eliminated 
from PL/I, while functions effective for raising the re¬ 
liability of software, such as argument check, are intro¬ 
duced 

• Labeled constants are also accepted for the calculation 
type data 

• Protection for the protected variables is given by dec¬ 
laration 

• Debugging at the source program level, such as the 
tracing of program flow and variables, is facilitated by 
debugging statements 

• Re-entrant function 

• Process input/output subroutines 

• Task control function. 


Although we have several kinds of computers in the distrib¬ 
uted system, programmers are freed from learning many 
kinds of languages. 

TPL statements also support the functions necessary to 
succeed: 

• Data transfer between main memories of two computers 
apart 

• “Send” and “receive” data between tasks each of 
which resides on a different computer 

• Data transfer to and from any input and output devices 
through data high-way. 


PERFORMANCE TEST 

The software life-cycle for our process control systems 
are composed of minor elementary cycles including require¬ 
ments definition, design, implementation, test and mainte¬ 
nance cycle. There are milestones in the cycle of which two 
are the ones treated in this section. 

One of the milestones is intended to validate some se¬ 
lected values of the performance before deciding system 
architecture and software configuration. 

In the other milestone, during test cycle, whole values of 
performance are measured and tested. 

The response time of each task exceeded the required 
limit is liable to cause damage to the plant equipment and 
decrease the product quality. In order to avoid these cases 
occuring both in normal operation and abnormal states, de¬ 
signing optimal arrangement of hardwares and softwares are 
requisite. The first milestone is intended to validate response 
time of designed system and to find optimal system struc¬ 
tures. Predicting response time by calculation is very diffi¬ 
cult. So, digital simulation using SIMSCRIPT is adopted as 
an automated analysis tool, in this milestone. 1 

Description is made here about an automated analysis tool 
for rolling mill computer control systems. 


Process modelling 

The modelling for simulation is shown in Figure 3. A, B, 
and C, are entities which simulate outside objects commu- 



Figure 3—Process model 
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Figure 4—Task procedure model for a computer 


nicating with the computer, e.g., the load cell and the hot 
metal detector. 

The modelling for tasks inside of the computer is made as 
shown in Figure 4. The paths which activated tasks take 
through the following resources are simulated. 

• Scheduling queue 

• Bulk to core transfer channel 

• Processor 

• Peripheral device control 


% 



Figure 5—A result of simulation 


The maximum and average values of each of these items are 
obtained. 


The result of analysis 


The tasks are terminated after passing through some of these 
resources. 


Input data to the analysis tool 

All items which are necessary to model hardware config¬ 
uration and their abilities, all of task names and their attri¬ 
butes such as average execution times, average frequencies 
of I/O requests, average sizes of input/output data are inputs 
to the automated analysis tool. 

Average intervals by which interrupts from the outside 
world occur are also input data to the analyzer. 

The analyzer is described with Simscript. 


Output data obtained 

The following data are obtained as the output of the ana¬ 
lyzer. 


An example of results of analysis is shown in Figure 5, 
Figure 6 and Table II. 

In order to obtain the peak load condition in the worst 
case load, the simulation for some duration of time ranging 
from 20-25 minutes is required. 

According to our experience, a system can be acceptable 
as having sufficient response if it has an average processor 
utilization rate below 30 percent and the utilization rates of 
channels below 70 percent. 

Table II shows the accuracy of the analysis tool in the 
values of response times. The values given by the simulator 
are compared with the ones measured in the actual operation 
after the installation. Figures 5 and 6 are the examples of 
the plotter output. 

Utilization rates of the processor corresponding to time 
are drawn in Figure 5. In the system related with Figure 6, 
the response time of an activity is required. The activity 
consisting of six tasks as shown in the figure is activated at 
time O and terminated at time T. The figure shows that T 
is the required response time. 


• Main memory occupancy rate 

• Bulk memory transfer channel utilization rate 

• Arithmetic unit utilization rate 

• System load rate 

• Response time of each task 

• Execution time of each task 

• Bulk transfer time of each task 

• Device control time of each task 

• Waiting time for execution 


TABLE II.—A Comparison of Simulation and Measured Response Time 


TASK 

AVERAGE OF RESPONSE TIME 

SIMULATION 

ACTUAL 

ROLLER TABLE 
CONTROL 

02 7 s e c 

0.2 0 s e c 

MILL CONTROL 

0.1 5 s e c 

0.2 0 s e c 

MILL SET UP 

1.0 1 s e c 

1.0 0 se c 












A Distributed Processing System 


1277 


I/O CONTROL TASK 


TRANSFER f 

CPU 

CRT DISPLAY EDITING TASK 

TRj 

QU] 

ill_ 

1 1 

1 

CPU 


DATA TRANSMISSION TASK 

TRANSFER 

i 

H 

i 

CPU | 

1 

1 

1 

SHEAR SET UP TASK 


TRANSFER j 

1 

CPU 

i/o ; 

i' i 

i i 

i i 

i i 

i 

_i_ 

TRACKING DISPLAY TASK 

1 

1 

1 

TRANSFER_ CPU i 

[QUEUE | 1 

I/O i 

__ 1 _ 

SHEAR LINE 

OBJECT TRACKING TASK 

T 

1 

RANSFER | 

! 

CPU 


TASK TO 

CONSTITUTE 

AN ACTIVITY 

0 

START 

100 200 3d0 401) 500 600 

«- TOTAL RESPONSE TIME -- 

THE TERMUS 

7(fo 

T 

LATE 


ACTIVITY THE ACTIVITY 

Figure 6—A result of simulation 


TIMECmse c) 


The second milestone is allocated in the testing cycle. We 
have in-house computers which are the same types as the 
computers to be shipped to customers. These in-house com¬ 
puters are connected with a data high-way. Distributed sys¬ 
tems to be shipped are able to be simulated with some 
modifications just for the purpose of test, by this in-house 
testing facility. We have another computer which we call a 
dynamic simulator. The programs run in this computer sim¬ 
ulate industrial equipment or dynamic behaviors of industrial 
processes. The dynamic simulator is connected to each one 
of the above related in-house computers. The programs im¬ 
plemented in the previous cycle are loaded to in-house com¬ 
puters and simulated execution is made. 

The structure of the dynamic simulator 

The plant processes are classified into continuous pro¬ 
cesses, batch processes, and dynamic characteristics of in¬ 
dividual elements. These are modularized. All kinds of pro¬ 
cesses can be simulated by combining selected modules. 

For continuous process, such a function module that ob¬ 
tains an output from a plural input data through logical or 
arithmetic procedure is provided. 

Elementary modules for the batch process, in case of a 
rolling mill process for example, simulate roller tables, roll¬ 
ing mills, transfers and shears according to rolling speed. 


length, width, and rolling directions. It also simulates sensor 
outputs and the metal detector signals. 

How to combine these elementary modules are pro¬ 
grammed by using a macrolanguage. 

Execution of dynamic simulation 

Each of the elementary modules are invoked in the timing 
generated by the scheduler, and are decoded and executed. 

At the execution stage, simulated process data are sent 
out to the in-house computers, and outputs from the in- 
house computer are received by the dynamic simulator. 

The simulated process data include, for example, metal 
detector ON/OFF, rolling speed, rolling torque and slab 
positions. 

The outputs from the in-house computers include rolling 
directions, mill roll gaps, and mill speed set up values. Over 
80 percent in number of test items can be validated by this 
off-site testing facility. 


TYPICAL APPLICATIONS 2 ’ 3 

From among the currently installed distributed processing 
systems, some latest examples are picked up with brief 
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description. The selected fields are: 

(1) Iron and steel industry 

(2) Water supply and sewage works 

Iron and steel industry 

This industry in Japan has hitherto pursued high produc¬ 
tivity and high yield rate in regard to the products with the 
targets: 

• Enlargement of plant scale with continuity of operation 

• Quantitative enlargement of products; diversification of 
species for products 

• Speed up of material handling 

• Energy saving. 

Success will not be obtained unless we have a total system 
covering from the management level throughout the works 
to product planning, process automation and minor physical 
control. 

As an example, herein cited is a large-scale total auto¬ 
mation system for a rolling mill. The system shown in Figure 


7 is a four-level hierarchy system, comprising the following: 

A-level—central on-line system 

B-level—mill area on-line system 

C-level—process control system 

D-level—local controller using a microprocessor or a min¬ 
icomputer 

These subsystems, linked with each other through looped 
data high-ways, perform high-speed exchange of informa¬ 
tion. 

The process control system is composed of four large- 
scale process computers with allotted tasks as follows: 

No. 1 computer—mill line area, 

No. 2 computer—shear line area. 

No. 3 computer—refining line area, 

No. 4 computer—backup for No. 1, 2, 3. 

The peripheral equipment for these computers, totaling 80 
units installed at every key point, are controlled through the 
looped data high-way surrounding the plant. 

The process control system, based on the information 
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Figure 7—Block diagram of steel mill computer system 
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exchange with the production control system of the B-level, 
guides and monitors the total processes covering from the 
slab yard, which is the entry of the mill line, to the stock 
yard, which is the exit of the refining line, calculates the 
optimum setup value, and gives control information to local 
controllers of the D-level. 

The principal functions of the process control system may 
be summarized as follows: 

• Object tracking 

• Slab yard mapping 

• Furnace temperature control 

• Mill Setup 

• Cutting schedule calculation and shear setup 

• Inspection and classification automation 

• Refining automation 

• Product-yard mapping 

® Shipping schedule 

• Automatic handling cf objects through total line (roller 
tables, cranes, transfers and other material handling 
equipment). 

The purpose of introducing this system lies in: improve¬ 
ment of productivity and yield rate, labor saving and im¬ 
provement of process control accuracy in the range of order 
receiving to product shipping. This purpose is now being 
accomplished with excellent results. 

Water supply and sewage works 

For water supply systems, effective utilization of limited 
water resources and treatment of waste water have become 
an important social problem in Japan to be solved quickly. 

On the other hand, to meet expanding of area to cover a 
growing complexity of equipment, increasingly higher tech¬ 
niques of operation are required. Shortage of operators as 
well as rise of cost of operation are also oppressing the 
management of projects. 

Under such conditions, the introduction of computers is 
now indispensable to water supply facilities. 

Figure 8 shows the total supervisory control system in a 
latest large-scale sewage works. 

The supervision is conducted by the duplicated mega-min¬ 
icomputer in the central control room, while the control is 
performed by the distributed systems employing a digital 
microcontroller or a soft-wired sequence controller linked 
with each other through the looped data way. 

The major functions of this automation are: 

Electrical power distribution control 
In-house power generation control 
Waste-water level control 
Sewage pump control 
Aeration control 
Recirculation sludge control. 

The automation for these objects enables the operation of 
100 DDC loops and speed control of 20 pumps, etc. This 



Figure 8—Block diagram of the centralized supervisory control system for 
a sewage works of Yokohama City 


system is making great contribution to the rational operation 
of the largest sewage works in Japan that treats waste water 
of one million tons a day. 

CONCLUDING REMARKS 

There are inherent requirements to process control com¬ 
puter systems that are input and output capabilities both to 
physical objects and to human objects, restricted response 
time, safety consideration both to human beings and equip¬ 
ment, and reliability. 

It is thought that the distributed system architecture has 
more capabilities to meet those requirements. 

The major components to constitute the system are data 
high-ways, micro-controllers, mini-computers including 
mega-mini and remote scanners. These components are 
more or less modularized in the aspect of both hardware 
and software. The modularity will increase flexibility of de¬ 
sign, quick shipment capability and good maintenance ca¬ 
pability. 

The concept is believed to be compatible with more com¬ 
plex system requirements in the future. 
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